Update Perl 6 samples

[hoelzro]

Add some more Perl 6 examples so the classifier does a better job.

[hoelzro]

Fixed the pull request to use fewer files, and to actually pass the Travis build. =) I'm also curious about the policy on how to include the licenses for sample files, as well as whether or not it would be appropriate to include a Perl 6-specific heuristic in lib/linguist/heuristics.rb

[AnAshy]

Thanks for checking the program for me

[arfon]

@hoelzro what sort of Perl 6 specific syntax were you thinking of for lib/linguist/heuristics.rb ?

[hoelzro]

@arfon So for Perl 6, we have two heuristics for disambiguating from Perl 5 according to the spec:

• Checking for use v6
• Checking for a module, class, or role declaration

In pygments, I check for these before any other non-empty, non-comment line appears in the file. If they appear, the file is considered Perl 6. Otherwise, it's considered Perl 5.

[hoelzro]

Has anyone had a chance to look at this?

[zhuomingliang]

+1 to merge

[skids]

👍

[chenryn]

+1

[hoelzro]

@arfon Is there a way this could get merged within the next week? A lot of Perl 6 code out there is being improperly highlighted, and I saw you folks intend to cut a release sometime next week. It would be really great if this could make it in!

[timo]

👍

[kanatohodets]

+1!

[hoelzro]

Is there anything I can do to help get this merged? The Perl 6 community is still pretty small, but it would make all of us happy!

[hoelzro]

Updated with fresh commits.

[hoelzro]

I refreshed the commits again, this time removing the update to samples.json, because I figured you folks will regenerate it anyway!

[hoelzro]

Updated against upstream, and included tm_scope for the new Textmate stuff.

[arfon]

@hoelzro - I'd like to get this merged but I'd really rather not make this change to the samples.rb file. How would you feel about removing this and adding in a Perl6 specific heuristic here:

https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb#L60-L68

use v6 seems like it would do a good job at disambiguating Perl 5 from 6?

[hoelzro]

@arfon That sounds good to me! The heuristic I use for pygments (as well as my Vim setup) involves a little more work (checking for class/module declarations in addition to use v6); should I add that to the linguist heuristic as well? Or should I just keep it simple for now?

[arfon]

I'd rather keep things as simple as possible if that's ok?

[timo]

to be honest, use v6 (or v6 or 6 alone) is only used very rarely so far. on the other hand, having a class or module declaration is a very good sign for perl6 while having a package declaration is a clear sign against perl6 and those two occur much more commonly than "use v6" does ...

[arfon]

Ok - add in what you think is best and let's have a discussion around the code in the pull request.

[hoelzro]

@arfon I added a heuristic for detecting Perl 6 - let me know if it's too complicated. It doesn't work entirely at the moment because samples.rb detects a shebang line, sees 'perl', and assumes it's Perl 5. I think that original change we were talking about yesterday may be necessary. I still have to rebase this code on top of master, but I wanted you to be able to look at it sooner rather than later.

[pchaigno]

@hoelzro From what I understand you removed the comments before applying the heuristic. Is it really common to find use v6 or class/module declarations in Perl comments? Using the heuristic directly would make things a lot simpler.

[arfon]

@hoelzro From what I understand you removed the comments before applying the heuristic. Is it really common to find use v6 or class/module declarations in Perl comments? Using the heuristic directly would make things a lot simpler.

Yeah - agreed. I'd much rather keep to 'simple' regexes rather than having custom pre-processing steps for language files such as the strip_pod you've made here.

[hoelzro]

Yeah, I'll get rid of that. If things start getting misclassified, I can always add it back!

[hoelzro]

@arfon @pchaigno Any thoughts on the shebang detection?

[hoelzro]

Ok, now I remember why stripping the POD/comments are important (and there's actually a bug in what's available now). The Perl 6 spec says that if the first non-comment line is a use v6 declaration, or a class/module declaration, then the file should be considered Perl 6; otherwise, it should be considered Perl 5. My code should be breaking out of the loop as soon as it sees a non-comment, non-declarative construct. If it didn't, it may misidentify files. For example, this is valid Perl 5 that could get misclassified at Perl 6:

use Moops;
class Person {
  has name => ( is => 'ro', isa => Str );
}

If we don't detect POD in particular, things like the following could be missed:

=pod

=head1 NAME

FancyRole

=head1 SYNOPSIS

class MyClass with FancyRole { ... }

=cut

If you still think this is too complicated logic for the linguist heurstics, maybe Perl 6 vs Perl 5 should rely on the bayesian classifier.

[pchaigno]

@hoelzro So you mean that module, class, role, enum and grammar are valid keywords in Perl 5? In that case we could just use use v6; and rely on the Bayesian classifier if it is not present.

For the shebang detection, do you mean that it doesn't work (gets detected as Perl 5) if you add interpreters: - perl6 to the Perl 6 entry in languages.yml?

[hoelzro]

Perl 5 can extend its list of keywords; for example, Moops adds the class and role keywords. That's why my heuristic for pygments stops at the first non-comment line. use v6 isn't too commonly used yet. As far as the shebang goes, last I checked it didn't work with Perl 6.

[pchaigno]

Any other construction specific to Perl 5 that we could use?

[hoelzro]

@pchaigno I can't think of anything that would be in a high percentage of Perl 5 programs; just use strict, but one could also have a strict module in Perl 6 as well. But that's probably the best metric I can come up with right now.

[pchaigno]

Sorry, I meant specific to Perl 6 but I guess Perl 5 keywords can also be used for heuristics.

[hoelzro]

Well, I for one would really like the community to get on board with use v6, but I don't know if that'll happen any time soon. =) Some language features like dynamic variables (ex. $*VAR) may help, or use of the dot operator rather than the arrow operator for method calls.

[pchaigno]

So you mean that Perl 5 doesn't have the following Perl 6 constructions?

$somevar.somemethod();
$*somevar = $somethingelse;

[hoelzro]

Correct. Granted, the former is valid Perl 5 (it just means concatenate $somevar to somemethod()), but I think that most Perl 5 code out there puts spaces around the . operator, whereas that would be unlikely in Perl 6.

[arfon]

OK, so while it's not as widespread as we might like, use v6 seems like a definitive (albeit not that common) identifier for Perl6?

I'd suggest using that for now unless there are better options.

Also, would you mind bringing this up to date with master @hoelzro? The heuristics code has been significantly reworked this last week.

[hoelzro]

@arfon Done! I've also changed the heuristic to just look for use v6; if that becomes an issue, we can raise it again.

[arfon]

Is it possible to find a TextMate or Sublime grammar that supports Perl6? I found this one but I'm not sure if it'll work straight out of the box: https://github.com/ashgti/perl-tmbundle Also, it's missing a license.

[hoelzro]

@arfon That was the one I was recommended; however, does it interfere with https://github.com/textmate/perl.tmbundle?

[arfon]

Ah yes, that will conflict as they're defined with the same name. How well does the standard Perl syntax highlighter do on Perl6 files?

It might be worth trying out https://github.com/textmate/perl.tmbundle on https://lightshow.githubapp.com

[hoelzro]

I don't think it would do very well; Perl 6 is a very different beast from Perl 5. I know that the pygments and vim highlighters for Perl 5 don't do a very good job with it!

[pchaigno]

Should we ask https://github.com/ashgti/perl-tmbundle if they can change their scope name?

[arfon]

Yes, that's a good idea.

[arfon]

OK, I'm going to merge this without a tm_scope at this stage. We can one in later once this naming issue is resolved.