Is there any tool to help convert gawk scripts to miller DSL?

[pjfarleyiii]

I have a set of fairly complex gawk programs (~300 to ~425 lines each) for manipulating financial CSV files (mostly for generating bank statement extracts to CSV data files). I wondered if anyone is aware of any tool or project that might help speed up a conversion of those gawk scripts to miller DSL.

Obviously a manual conversion is always going to be more accurate, but it is a somewhat tedious process for large scripts so I wondered if there was any help available to speed up a conversion.

[johnkerl]

Oh my -- I for one have never done this.

A fully automatic transpile would be a Big Project. But for various conversion projects in my life -- e.g. porting something from one language to another -- I often use a mixed automated/manual approach. Namely have something simple like some one-off sed scripts to do 90% of the repetitive edits; then, manually edit the rest. This often ends up being a good balance of time expenditure to results ...

Can you share a few of your scripts??

[pjfarleyiii]

Sorry, most of these scripts are way too specific to my personal spending history to let me feel comfortable sharing them. An awful lot of the code is devoted to correcting my bank's atrocious mangling and obfuscating of the actual vendor name that I paid for a good or a service so that I can track specific types of spending by assigning a category based on the name. The variety of corrections that I have to make is ridiculously large but very specific to me personally.

The process of scanning the text of a bank statement extracted from a bank PDF would probably be something shareable, but would be very specific to my bank's statement formats. Unfortunately I originally integrated most of that extraction logic interspersed with and sometimes interacting with a lot of the name decoding logic, so separating the two distinct sets of logic is the first task I would need to accomplish before I could share anything.

Your advice on automating a lot of the repetitive edits first and then manually performing the harder parts is good, and I will try to follow that advice.

It would be interesting if gawk could be persuaded to export an AST tree in some reasonable text format to be post-processed by a language conversion tool able to interpret AST's, but that's a non-starter for now. I am far from experienced enough in disentangling the yacc/bison source parts from the main gawk sources to set up something as sophisticated as that would be.

[johnkerl]

OK great @pjfarleyiii !! :)

FWIW this is how I converted big slabs of Miller from C to Go -- e.g. replacing printf with fmt.Printf and so on, and letting the compiler duly tell me what else I needed to do :) 🤞

[janxkoci]

The syntax of miller is generally not that far from gawk. But there are a few places with important differences.

For example, miller uses names when referencing to columns, such as $date, rather than gawk's numeric $1 style. You could use $[[[1]]] in miller for automatic conversion of such column references, but you are losing one of the main benefits of miller. Getting named column references will require manual translation, as the names depend on your data.

And it will be similar with other concepts - automatic conversion will only allow crude translation, but will most likely not use any fancy features of miller. But if you know what needs to be changed, it should be easy with a sed script and a few manual tweaks..

The more important question is - do you need to translate the scripts? Does miller provide any large benefit over gawk? As I code in both, I suspect the answer might be a nope. 😉