[feature] manipulate captured variable in fix mode

[HerringtonDarkholme]

Suggestion

⭐ Suggestion

I think it is necessary to support string manipulation in fix mode. Reference #430

It is impossible to match some patterns in some languages. String manipulation is a must to support example like get the inner content of a generator expression in Python.

📃 Motivating Example

Use ast-grep to change ESM *.mjs to CJS require("*").

rule: 
  pattern: import {$BINDING} from "$PATH"
transform:
   NEW_PATH:
     substring:
       endChar: -4
       source: $PATH
fix: var $BINDING = require("$NEW_PATH").binding

Proposal

New Field in YAML Rule

We will add a new field transform in YAML rule.

The API will use YAML to encode string operations.

Pro:

• No cryptic oneliner string manipulation
• Not inventing a new script-like language
• Easier to implement
• Support nested string manipulation

Cons:

• Learning YAML is not easier than learning script-lang
• More verbose than string DSL
• Easier to create heavily nested transformation like Hadouken

Substring

NEW_VAR: 
  substring: 
    startChar: 1
    endChar: -1
    source: $VAR

Replace

NEW_VAR:
  replace:
    replace: regex
    by: replacement
    source: $VAR

Convert

Similar to
https://www.typescriptlang.org/docs/handbook/2/template-literal-types.html#intrinsic-string-manipulation-types

NEW_VAR: 
  convert:
    toCase: upperCase # or lowerCase, capitalize, camelCase, snakeCase, pascalCase, kebabCase
    source: $VAR

You can specify separator by separatedBy option in convert.

NEW_VAR: 
  convert:
    source: $VAR
    toCase: kebabCase
    separatedBy: [underscore] # or caseChange, space, dot, slash, dash

Deletion (IMPLEMENTED as replace)

Similar to replace but leave replace as empty.

NEW_VAR: 
  replace:
    replace: regex
    by: ""
    source: $VAR

Concat (NOT IMPLEMENTED)

NEW_VAR: 
  concat: 
     sources: [$VAR1, $VAR2, "plain string"]

[HerringtonDarkholme]

I will only support string slicing in my first implementation.

[suyanhanx]

$VAR/regex/replace/

Is this replace a keyword or we will replace the matched part with this content?

[HerringtonDarkholme]

@suyanhanx replace will be the actual replacement string. Example to change .js to .cjs

$PATH/\.js$/.cjs/

[suyanhanx]

The regex and replace parts may be difficult to distinguish.

[HerringtonDarkholme]

@suyanhanx Any suggestion to make them clear?

[suyanhanx]

It's actually difficult to decide which character to choose as the separator. Can we provide an additional configuration option to allow users to choose their own separator?

Of course, we could also default to a common separator, such as # or %.

[HerringtonDarkholme]

Can we provide an additional configuration option to allow users to choose their own separator?
It will introduce community splitting and additional implementation complexity.

I remember Vim has it and it is probably too nerdy to most modern devs.
Also, changing separate character does not solve the order problem.

[suyanhanx]

Alternatively, we could change the format by making it like a function call or dividing it into different configuration options.

[HerringtonDarkholme]

Old Proposal, withdrawn.

rule: 
  pattern: import {$BINDING} from "$PATH"
transform:
   NEW_PATH: PATH[..-3]
fix: var $BINDING = require("$PATH").binding

transform is a dictionary of key-value pairs. Key is the new meta variable name.
Value is the transformation written in string manipulation syntax.

String Manipulation Syntax

Replace

$VAR/regex/replace/

Deletion

Similar to replace but leave replace as empty.

$VAR/regex//

Concat

$VAR1$VAR2

Transform

Similar to
https://www.typescriptlang.org/docs/handbook/2/template-literal-types.html#intrinsic-string-manipulation-types

$VAR.upperCase
$VAR.lowerCase.capitalize # multiple transformation can be chained

Substring

Negative index is supported.

$VAR[1..-1]

💻 Use Cases

change file extension from .js to .cjs / .mjs. See #315
support rewrite python generator syntax. See #430

[HerringtonDarkholme]

Added in 0.6.3