Update script/licensed usage to accommodate upstream changes

[lildude]

The next version of licensed will cause a breakage in our ability to add grammars, particularly around grabbing the license. This PR fixes that.

Description

licensed changed the method sources are defined in licensee/licensed#33. As we use a self-defined Filesystem source, we don't get the upstream changes for free so need to make changes ourselves else we'll get failures like this when users attempt to add or update a grammar:

$ script/add-grammar https://github.com/newgrammars/m3
Checking docker is installed and running
$ docker ps
Registering new submodule: vendor/grammars/m3
$ git submodule add -f https://github.com/newgrammars/m3 vendor/grammars/m3
$ script/grammar-compiler add vendor/grammars/m3
Confirming license
$ script/licensed --module /Users/lildude/github/linguist/vendor/grammars/m3
  > Caching licenes for linguist:
  > /Users/lildude/github/licensed/lib/licensed/command/cache.rb:21:in `block (3 levels) in run': undefined method `type' for Licensed::Source::Filesystem:Class (NoMethodError)
  > 	from /Users/lildude/github/licensed/lib/licensed/command/cache.rb:20:in `map'
  > 	from /Users/lildude/github/licensed/lib/licensed/command/cache.rb:20:in `block (2 levels) in run'
  > 	from /Users/lildude/github/licensed/lib/licensed/command/cache.rb:17:in `chdir'
  > 	from /Users/lildude/github/licensed/lib/licensed/command/cache.rb:17:in `block in run'
  > 	from /Users/lildude/github/licensed/lib/licensed/command/cache.rb:12:in `each'
  > 	from /Users/lildude/github/licensed/lib/licensed/command/cache.rb:12:in `flat_map'
  > 	from /Users/lildude/github/licensed/lib/licensed/command/cache.rb:12:in `run'
  > 	from script/licensed:53:in `<main>'

This PR fixes this by slightly modifying and simplifying our self-defined Filesystem source.

$ script/add-grammar https://github.com/newgrammars/m3
Checking docker is installed and running
$ docker ps
Registering new submodule: vendor/grammars/m3
$ git submodule add -f https://github.com/newgrammars/m3 vendor/grammars/m3
$ script/grammar-compiler add vendor/grammars/m3
Confirming license
$ script/licensed --module /Users/lildude/github/linguist/vendor/grammars/m3
Updating grammar documentation in vendor/README.md
$ bundle exec rake samples
$ script/sort-submodules
$ script/list-grammars
$
$ git status
On branch lildude/accom-new-licensed
Your branch is up to date with 'origin/lildude/accom-new-licensed'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   .gitmodules
	modified:   Gemfile
	modified:   grammars.yml

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	vendor/grammars/m3/
	vendor/licenses/grammar/m3.txt

no changes added to commit (use "git add" and/or "git commit -a")
$

This will mean we need to enforce a minimum requirement for Licensed once a new release has been made that includes the changes from licensee/licensed#33 and licensee/licensed#41. I'll add this to this PR once the release has been made.

@jonabc do you see any problems with the approach I've taken here? Or maybe we should be changing the method we use entirely so we don't have to do this if licensed changes like this again in future?

Checklist:

• I am adding new or changing current functionality
  • I have added or updated the tests for the new or changed functionality.

[jonabc]

@lildude The changes look great.

Or maybe we should be changing the method we use entirely so we don't have to do this if licensed changes like this again in future?

I haven't tested this, but you might be able to remove the custom source entirely and use the manifest source type. You'd need a manifest file listing each dependent project, but I think it'll work if you have a single entry for each grammar, that is the path to the vendored grammar folder.

example manifest.json, yml also works 👍

{
  ...
  "vendor/grammars/llvm.tmbundle": "llvm.tmbundle",
  "vendor/grammars/m3": "m3",
  ...
}

Maintaining the manifest file would need to happen locally. Looks like that could be done with scripting via a re-mapping of grammars.yml.

[lildude]

Thanks @jonabc. That's an interesting approach. I'll investigate that avenue when I've got more time. For the moment, lets stick with what we've got.

@Alhadis @pchaigno see any problems with this PR?

[jonabc]

@lildude FYI, licensed-1.1.0 has been released going along with these changes

[lildude]

This is looking good now and ready for review.

I've updated the licenses whitelist in this PR too as a recent update to licensee has introduced a change that has resulted in a slight variation in the hash for four of the grammars, resulting in all the CI failures that have been occurring in the last 2 weeks.

[jonabc]

Looks great from the licensed side

[lildude]

Merging to get tests passing again.