Custom Expression name containing brackets ([ or ]) can break the UI

[flamber]

Describe the bug
When a Custom Expression uses a name containing brackets ([ or ]), then autocomplete dropdown stops working and forcefully trying to use that expression again in another expression will break the UI until browser refresh (or show "Something went wrong").

To Reproduce

1. Custom question > Sample Dataset > Orders
2. Custom Column formula 1 + 1 with the name MyCC [2021]
3. Summarize > Custom Expression > type Sum([ notice there's no field dropdown
   
4. Summarize > Sum of > pick the MyCC [2021]
5. Click the metric again, and click the title to go back to list of metrics and click Custom Expression - the UI is now broken and requires a browser refresh to function again (on upcoming 0.39 master, it will reset everything done in the Notebook and show "Something went wrong" in the lower-right corner)
   

There are a lot of browser console errors

Firefox: TypeError: e._wrapperState is null
Chrome: Uncaught Error: String currently can't contain brackets: MyCC [2021]      <-- hey, why didn't it tell me that in the UI?

Information about your Metabase Installation:
Tested 0.38.2 and master

Severity
Giving P2, since it breaks UI (or clears everything on master), even though it's an easy workaround - don't use brackets.
Are there other characters that could cause similar problem?

[howonlee]

confirming different-but-same brokenness for parens

[ariya]

This is currently the limitation of the lexer used in the Chevrotain-based parser, it does not understand a field reference that contains brackets or parentheses or other special characters (which must be escaped), e.g. [MyCC \[2021\]].

[ariya]

Now that I think about this problem, maybe the solution is to prevent the use of these special characters as the name of any custom column. It's going to cumbersome anyway to use a name that has brackets, even if character escaping is supported.

Thus, the plan is: if the custom column name contains one ore more special characters, show an error message and don't let the user continue.

Does that sound like a good compromise?

[howonlee]

Doesn't this one affect non-custom columns too? Quote delimited columns in postgres can have anything - brackets, emoji...

[flamber]

@ariya Sounds great, but as @howonlee mentions - this can also happen via the display name changed in Admin > Data Model.
Just tried Subtotal [2010] - then no suggestions are shown, when trying to summarize and console borks with:
Error: String currently can't contain brackets: Subtotal [2010]