Only a subset of syntax is highlighted

[zats]

Hi, I was playing around with initial setup of the library.
It seems like the library recognises only a subset of the syntax (while the same file opened in TextMate is showing the full syntax highlight – not sure if it's relevant)

This is the code I used (more or less copied from the readme, slightly modified for Swift 2):

let samplePath = bundle.URLForResource("Sample", withExtension: "swift")!
let sample = try! String(contentsOfURL: samplePath)

let languagePath = bundle.URLForResource("Swift", withExtension: "tmLanguage")!
let languageDictionary = NSDictionary(contentsOfURL: languagePath) as! [NSObject: AnyObject]
let language = Language(dictionary: languageDictionary)!

let themeURL = bundle.URLsForResourcesWithExtension("tmTheme", subdirectory: nil)!.first!
let themeDictioanry = NSDictionary(contentsOfURL: themeURL) as! [NSObject: AnyObject]
let theme = Theme(dictionary: themeDictioanry)!

let parse = AttributedParser(language: language, theme: theme)
let attributedString = parse.attributedStringForString(sample)

Following is the playground (please rename to .zip and extract).

Is anything I'm doing look obviously wrong?

And a follow up question: is there a smarter way to figure out the language used for a file than the file extension? Extensions might be used by several languages: .h file might be either Objective-C or plain C; while .m is used for both Objective-C and Matlab.

Thanks a lot!

[dehlen]

I got the exact same problem. Only parts are highlighted. Other are recognized but the attributes array is nil.
Could you please have a look into it. Provided Playground by zats replicates the problem although my problem ist that LaTeX comments and commands like \command{} do not get highlighted with some exceptions (\begin{document}, \usepackage{}, \newcommand{}) is working. Would it help you if i provided my current setup also ?

Or do you think that this is a problem concerning the tmLanguage, tmTheme files ? When I open my test file in Textmate a lot more is colored than in my own NSTextView.

[dehlen]

Just had a look at this this morning and for me it looks like the library is working just fine. It seems like the plist(tmLanguage) file is missing entries and therefore the code is only party highlighted.

[zats]

I think I copied the bundle from the official repo. However I'll need to
check that
On Sat, Jul 25, 2015 at 11:30 AM David Ehlen notifications@github.com
wrote:

Just had a look at this this morning and for me it looks like the library
is working just fine. It seems like the plist(tmLanguage) file is missing
entries and therefore the code is only party highlighted.

—
Reply to this email directly or view it on GitHub
#5 (comment).

[dehlen]

Sure please report back. I only had a look at the Latex.tmLanguage file and it occured to me that there was no pattern for highlighting comments for example.

[zats]

Yeah, as I suspected, I just copied the language file from the official
repo (I played around with swift definition as you can see from the
screenshots).
Link to the playground includes my setup (except for the SyntaxKit that has
to be part of the workspace for the playground to work)
Please let me know if I can provide any more information to help understand
the issue better.
Thanks!

On Sat, Jul 25, 2015 at 11:43 AM David Ehlen notifications@github.com
wrote:

Sure please report back. I only had a look at the Latex.tmLanguage file
and it occured to me that there was no pattern for highlighting comments
for example.

—
Reply to this email directly or view it on GitHub
#5 (comment).

[Orion98MC]

Hi,

Same for me.

I have downloaded Swift.tmLanguage from the tm gh repo as well as the twilight theme.
Only storage.type.swift and support.function.swift are tokenized

Tried a different tmLanguage, like markdown => No recognized pattern at all when parsing a md file

Let me know if I can help in any way.

Regards,
Thierry

[Orion98MC]

Adding to my comments:

The following Language initialization log is taken from Language() for Swift.tmLanguage.

Here is an excerpt of the plist parsing which results from the execution of the "repository" parsing block at Language.swift:34:

The log shows how the Swift.tmLanguage 'repository.optional-type' keyPath is parsed:

Pattern:
    name: Optional("meta.optional.swift")
    Match: Optional("\\b(Optional)(<)")
    Begin: nil
    End: Optional("(>)")
    BeginCaptures: CaptureCollection(captures: [2: SyntaxKit.Capture(name: "punctuation.definition.optional.begin.swift"), 1: SyntaxKit.Capture(name: "support.type.optional.swift")])
    EndCaptures: CaptureCollection(captures: [1: SyntaxKit.Capture(name: "punctuation.definition.optional.end.swift")])
    1 Pattern(s): [
           Pattern:
            name: nil
            Match: nil
            Begin: nil
            End: nil
       ]

On the other hand, the plist itself for the same keyPath is:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>beginCaptures</key>
    <dict>
        <key>1</key>
        <dict>
            <key>name</key>
            <string>support.type.optional.swift</string>
        </dict>
        <key>2</key>
        <dict>
            <key>name</key>
            <string>punctuation.definition.optional.begin.swift</string>
        </dict>
    </dict>
    <key>end</key>
    <string>(&gt;)</string>
    <key>endCaptures</key>
    <dict>
        <key>1</key>
        <dict>
            <key>name</key>
            <string>punctuation.definition.optional.end.swift</string>
        </dict>
    </dict>
    <key>match</key>
    <string>\b(Optional)(&lt;)</string>
    <key>name</key>
    <string>meta.optional.swift</string>
    <key>patterns</key>
    <array>
        <dict>
            <key>include</key>
            <string>$self</string>
        </dict>
    </array>
</dict>
</plist>

I have no knowledge of the tmLanguage format but I noticed that the patterns keyPath within the repository.optional-type keyPath is translated as an array of 1 pattern with all nil values which seems to be a false interpretation of the plist data.

Now, I'm pretty sure many other keyPaths are not parsed correctly. Which may have significant side effects on the recognition of patterns when the parser runs.

Also the Language.patterns (for both Swift.tmLanguage and markdown) does contain many all-nil-values Pattern-s which seems to confirm that the parsing of tmLanguage files is not optimal

Thanks and regards,
Thierry

[soffes]

Ya there's definitely some outstanding bugs.

[dehlen]

Here is a definition of the textmate grammar: http://manual.macromates.com/en/language_grammars#language_grammars.

It looks like the include key is not parsed correctly in SyntaxKit. The different values a include key can hold are described in the link above. Is there any plan to fix this issue in the future ? It seems like this library was last updated quite a time ago.

Cheers,
David

[soffes]

This is amazingly helpful @dehlen! I didn't know this existed. I'll be using this library again in a project I'm working on in about 2 months, so I'll get started with some some tests and full support when I start using it again.

Sorry I've been slow to work on this everyone. Watch the linked video above if you want to know why :)

[dehlen]

You don't have to be sorry. Open-Source is sharing and caring :) I am myself a bit stressed at the moment writing my thesis. However if I'll find time I'll might have a look at the outstanding bug.

Keep up the amazingly good work.

[soffes]

Thanks <3

[aaronvegh]

The timing on this thread is really good. I came across this project last week and spent some time looking at the missing features in the Language.swift implementation. Updating it to support the new (?) syntax is going to be quite the challenge! I thought I'd take a crack at it, but it kicked my ass. So Sam, I will wish you all the best in getting it updated!