Expose tokenize() function

[thybzi]

Sometimes it's very useful to get tokenized CSS selector parsed by internal function tokenize()

Why not expose it for public use? Just like that, for example:

Sizzle.tokenize = tokenize;

[timmywil]

I don't have a problem with it. I'm curious, though. I'd be interested in hearing how you've used it in your own code.

[paulirish]

@thybzi what usecases are you looking to address? curious as well.

[josh]

I've wanted the same, maybe. And its nice that it shares the same tokenCache.

Though, with the ability to drop Sizzle in jQuery itself, I don't think I would rely on a sizzle only API in practice. Its awkward to access Sizzle through jQuery.find.tokenize if such an api was exposed.

It might be more interesting if something like CSS.tokenize was spec'd in browsers to complement utils like CSS.escape.

[thybzi]

Sorry for a long pause, somehow I missed GitHub answer notice.

I use this function to parse a CSS selector to components and detect classes, IDs and other possible "attribute-driven stuff".

That is used in my StyleDoc parser and showcase generator currently under development (insired with idea of https://github.com/Joony/styledoc, but fully rethought and rewritten).

I use a modified version of jQuery with tokenize() exposed to apply CSS modifier selector (e.g. .additionalClass) to HTML element (e.g. <span class="myClass"></span>), which produces something like <span class="myClass additionalClass"></span> in this basic example.

The code is a bit raw and ugly for now, here it is:

    styledoc.htmlApplyModifier = function(html, base, modifier)
    {
        var $wrapper = $('<styledoc-wrapper>').append(html);
        var $elem = $(base, $wrapper);

        if ($elem.length) {
            var parsed = $.find.tokenize(modifier).pop();
            var item,
                attrName,
                attrValue;
            for (var i = 0; i < parsed.length; i++) {
                item = parsed[i];
                attrName = null;
                attrValue = '';
                switch (item.type) {
                    case 'ID':
                        attrName = 'id';
                        attrValue = item.matches[0];
                        break;
                    case 'CLASS':
                        attrName = 'class';
                        attrValue = item.matches[0];
                        break;
                    case 'ATTR':
                        attrName = item.matches[0];
                        attrValue = item.matches[2];
                        break;
                    case 'PSEUDO':
                        attrName = item.matches[0];
                        break;
                }
                if (attrName === 'class') {
                    $elem.addClass(attrValue);
                } else if (attrName) {
                    $elem.attr(attrName, attrValue);
                }
            }
            html = $wrapper.html();
        }

        return html;
    };

[thybzi]

So, would you do this?

[gibson042]

One of the concepts I've been toying with and may propose in the near future is rearranging tokens between combinators for more efficient right-to-left filtering (e.g., .class#id:not(:disabled) → [ {type: "PSUEDO", …}, {type: "CLASS", …}, {type: "ID", …} ]). This is fine in the current codebase, but could be problematic if tokenize were exposed. Would such manipulation hinder your use cases?

[thybzi]

As for me, this shouldn't cause any problem, I just maybe replace pop() with shift(), if I'll need to :)

[dmethvin]

If you're planning to use this in some kind of plugin or other public code that other less-capable people might use, we'd want to be careful with this kind of change. It's always the danger in exposing these kind of internals.

[jokeyrhyme]

If this was available, I'd use it to allow my project to suggest optimisations for jQuery selectors: https://github.com/jokeyrhyme/jqlint
jokeyrhyme/jqlint#3

[thybzi]

So, would you do this?

I'd do it myself with a pull request, if someone tell my a right way (used by jQuery) to rebuild compiled/combined .js files when sources are changed. Such changes are also to be commited to repo, aren't they? I've found nothing about that at http://contribute.jquery.org/ :(

[timmywil]

okok, on it.

By the way @thybzi, instructions for contributing to Sizzle are in the README under "Developing with Grunt".

[timmywil]

Be aware this is still an undocumented function and is subject to change.

[bezborodow]

Is this missing now? jQuery 3.7.0.

$.find.tokenize('.test');
Uncaught TypeError: $.find.tokenize is not a function
    at <anonymous>:1:8

We use this in pie6k/jquery.initialize. We watch the DOM for mutations that match a selector, and being able tokenise the selector allows us to ignore certain mutations that the selector would never match against.

[mgol]

@bezborodow thanks for the info. I extracted your comment to a separate jQuery issue: jquery/jquery#5259.