tokenize(): produces inconsistent results

[layershifter]

Hey folks!

I am working on a plugin for :global() selector and using tokenize() there. I am not sure that it's a bug, but results are inconsistent:

tokenize(':global(body) .foo') 
// => [ ':', 'global', '(body)', ' ', '.foo' ]
//            👆 selector produces a single string
tokenize(':global([data-popper-placement]) .foo') 
// => [ ':', 'global', '([data-popper-placement])', ' ', '.foo' ]
//                      👆 selector produces a single string
tokenize(":global([data-popper-placement^='top'])")
// => [ ':', 'global', "([data-popper-placement^='top'", ']', ')' ]
//                                                       👆 closing bracket and a paren are separate strings

I also tried this string with namespace plugin:

serialize(compile(`:global([data-popper-placement^="top"]) .foo { color: red }`), middleware([namespace, stringify]))
// => [data-popper-placement^="top]) .foo[object Object]{color:red;}
//                                ^ brackets are closed properly ✔
//                                  ^ but "[object Object]" is definitely an issue ❌

Can you please clarify if it's a bug in tokenizer?

[thysultan]

The last one looks like a bug, not sure about the others, will look into it.

[layershifter]

@thysultan were you able to check if it is a bug? 😊

[thysultan]

The last one was a bug(fixed now on the latest patch), the rest you can manually recursively tokenize to get the desired depth of tokens.

[thysultan]

The last one was a bug(fixed now on the latest patch), the rest you can manually recursively tokenize to get the desired depth of tokens.