Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add USD lexer #1290

Merged
merged 52 commits into from
Dec 6, 2019
Merged

Add USD lexer #1290

merged 52 commits into from
Dec 6, 2019

Conversation

ColinKennedy
Copy link
Contributor

Ported from: https://bitbucket.org/birkenfeld/pygments-main/pull-requests/840/add-usd-lexer

Current Status

The code below was run against roughly 300 USD files that ship with Pixar's test suite. Everything is 100% working except for one exception case which I was hoping to get advice on.

Any time there's a multi-line string, it seems, non-alphanumeric characters register as an error.

e.g.

    int[] creaseLengths = [] (
        doc = """The length of this array specifies the number of creases on the
        surface. Each element gives the number of (must be adjacent) vertices in
        each crease, whose indices are linearly laid out in the 'creaseIndices'
        attribute. Since each crease must be at least one edge long, each
        element of this array should be greater than one.""")

The 's in this paragraph

As do characters like -> and \\

    inherits = </APISchemaBase>
    doc = """API schema to support discovery and publishing of lights in a scene.

    \\section UsdLuxListAPI_Discovery Discovering Lights via Traversal

    To motivate this API, consider what is required to discover all
    lights in a scene.  We must load all payloads and traverse all prims:

    \\code
    01  // Load everything on the stage so we can find all lights,
    02  // including those inside payloads
    03  stage->Load();
    04
    05  // Traverse all prims, checking if they are of type UsdLuxLight
    06  // (Note: ignoring instancing and a few other things for simplicity)
    07  SdfPathVector lights;
    08  for (UsdPrim prim: stage->Traverse()) {
    09      if (prim.IsA<UsdLuxLight>()) {
    10          lights.push_back(i->GetPath());
    11      }
    12  }
    \\endcode

Any advice on how to fix this would be very appreciated. Otherwise that issue, please let me know what else you'd like to have changed. Thank you!

@Anteru Anteru added this to the 2.6 milestone Nov 27, 2019
Copy link
Collaborator

@Anteru Anteru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can tell, adding this rule:
(r'"""(?:.|\n)*?"""', token.String), before the current string rule in usd.py seems to fix it. Can you please give it a try locally? With this rule, multi-line strings are matched correctly.

pygments/lexers/usd.py Outdated Show resolved Hide resolved
@ColinKennedy
Copy link
Contributor Author

ColinKennedy commented Dec 1, 2019

Hi @Anteru

Just checked on my end and everything is getting captured correctly! Thanks for your suggestion. I tried that regex rule but I put it in the wrong position so it didn't work in every case.

I've also added single-quote and double-quote variations and everything seems to work great now. Please let me know if you have any other notes.

Edit: Just realized that there's no tests right now. I'll add those shortly.

@ColinKennedy
Copy link
Contributor Author

Tests added and CI passes now. Thanks for your quick replies, @Anteru ! Please let me know if you'd like anything else changed

Copy link
Collaborator

@Anteru Anteru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks for the contribution!

@ColinKennedy
Copy link
Contributor Author

My pleasure! I don't have write access so I'll leave it to someone who does to handle the merge

pygments/lexers/compiled.py Outdated Show resolved Hide resolved
pygments/lexers/usd.py Outdated Show resolved Hide resolved
pygments/lexers/usd.py Outdated Show resolved Hide resolved
pygments/lexers/usd.py Outdated Show resolved Hide resolved
pygments/lexers/usd.py Outdated Show resolved Hide resolved
pygments/lexers/usd.py Outdated Show resolved Hide resolved
pygments/lexers/usd.py Outdated Show resolved Hide resolved
pygments/lexers/usd.py Show resolved Hide resolved
Copy link
Member

@birkenfeld birkenfeld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much, I'm happy now :)

@birkenfeld birkenfeld merged commit ff93d05 into pygments:master Dec 6, 2019
@ColinKennedy
Copy link
Contributor Author

Sweet, thanks for the merge. Really excited for the next Pygments release :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants