Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add yarn source #232

Merged
merged 3 commits into from
Dec 27, 2019
Merged

Add yarn source #232

merged 3 commits into from
Dec 27, 2019

Conversation

jonabc
Copy link
Contributor

@jonabc jonabc commented Dec 24, 2019

closes #31

This adds a yarn source that finds dependency information using a combination of yarn list --json and yarn info --json. The yarn info calls are parallelized to cut down on the runtime hit of calling the command on each dependency. Paths are constructed by looking for shadow: false children from yarn list output - these indicate that a dependency is nested and are found at <parent>/node_modules/<child>.

Otherwise it's very similar to the npm source, which I copied and modified to get to the current yarn source.

I've included the same configuration setting to pick up non-production dependencies that was added to #231. In testing, I found issues with this in versions <= 1.2.0 so it may not be dependable for older versions of yarn.

I've added CI actions for yarn@ 1.4.0 and latest versions, to test the minimum version installable with npm install -g yarn that supports yarn list --production along with the latest version.

/cc @krzysztof-pawlik-gat FYI from your input on #31 and #170

@jonabc jonabc merged commit d4a3f3e into master Dec 27, 2019
@jonabc jonabc deleted the yarn-source branch December 27, 2019 18:46
@sergey-alekseev
Copy link
Contributor

Great to see the yarn source to be added. I tested it locally and ran into different problems. E.g. the problem which occurred several consecutive times with different JSONs after running bundle exec licensed cache (see below):

$ bundle exec licensed cache
...
  yarn
bundler: failed to load command: licensed (/Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bin/licensed)
JSON::ParserError: 765: unexpected token at '{"type":"inspect","data":{"name":"ecc-jsbn","description":"ECC JS code based on JSBN","dist-tags":{"latest":"0.2.0"},"versions":["0.0.1","0.1.1","0.1.2","0.2.0"],"maintainers":[{"name":"quartzjer","email":"jeremie@jabber.org"},{"name":"rynomad","email":"nomad.ry@gmail.com"}],"time":{"modified":"2018-07-29T17:44:40.282Z","created":"2014-03-06T05:48:33.314Z","0.0.1":"2014-03-06T05:48:34.662Z","0.1.1":"2015-11-23T14:19:09.579Z","0.1.2":"2018-07-29T15:54:20.855Z","0.2.0":"2018-07-29T17:44:40.282Z"},"readmeFilename":"README.md","homepage":"https://github.com/quartzjer/ecc-jsbn","keywords":["jsbn","ecc","browserify"],"repository":{"type":"git","url":"git+https://github.com/quartzjer/ecc-jsbn.git"},"author":{"name":"Jeremie Miller","email":"jeremie@jabber.org","url":"http://jeremie.com/"},"bugs":{"url":"https://github.com/quartzjer/ecc-jsbn/issues"},"license":"MIT","users":{"mojaray2k":true},"version":"0.1.1","main":"index.js","dependencies":{"jsbn":"~0.1.0"},"gitHead":"d35a360352496721030da645e8054f07efc22487","scripts":{},"dist":{"shasum":"0fc73a9ed5f0d53c38193398523ef7e543777505","tarball":"https://registry.npmjs.org/ecc-jsbn/-/ecc-jsbn-0.1.1.tgz"},"directories":{}}}'
  /Users/sergey/.rvm/rubies/ruby-2.5.5/lib/ruby/2.5.0/json/common.rb:156:in `parse'
  /Users/sergey/.rvm/rubies/ruby-2.5.5/lib/ruby/2.5.0/json/common.rb:156:in `parse'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/sources/yarn.rb:78:in `package_info'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/sources/yarn.rb:44:in `block in packages'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/parallel-1.19.0/lib/parallel.rb:506:in `call_with_index'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/parallel-1.19.0/lib/parallel.rb:471:in `process_incoming_jobs'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/parallel-1.19.0/lib/parallel.rb:453:in `block in worker'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/parallel-1.19.0/lib/parallel.rb:444:in `fork'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/parallel-1.19.0/lib/parallel.rb:444:in `worker'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/parallel-1.19.0/lib/parallel.rb:435:in `block in create_workers'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/parallel-1.19.0/lib/parallel.rb:434:in `each'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/parallel-1.19.0/lib/parallel.rb:434:in `each_with_index'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/parallel-1.19.0/lib/parallel.rb:434:in `create_workers'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/parallel-1.19.0/lib/parallel.rb:373:in `work_in_processes'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/parallel-1.19.0/lib/parallel.rb:277:in `map'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/sources/yarn.rb:44:in `packages'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/sources/yarn.rb:14:in `enumerate_dependencies'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/sources/source.rb:65:in `cached_dependencies'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/sources/source.rb:48:in `dependencies'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:86:in `block in run_source'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/reporters/cache_reporter.rb:28:in `block in report_source'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/reporters/reporter.rb:106:in `report_source'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/reporters/cache_reporter.rb:26:in `report_source'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:81:in `run_source'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/cache.rb:26:in `run_source'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:64:in `block (3 levels) in run_app'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:64:in `map'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:64:in `block (2 levels) in run_app'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:57:in `chdir'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:57:in `block in run_app'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/reporters/cache_reporter.rb:14:in `block in report_app'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/reporters/reporter.rb:83:in `report_app'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/reporters/cache_reporter.rb:12:in `report_app'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:56:in `run_app'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:27:in `block (2 levels) in run'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:27:in `map'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:27:in `block in run'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/reporters/reporter.rb:61:in `report_run'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/commands/command.rb:22:in `run'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/cli.rb:83:in `run'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/lib/licensed/cli.rb:14:in `cache'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bundler/gems/licensed-8ad8e08e3152/exe/licensed:5:in `<top (required)>'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bin/licensed:23:in `load'
  /Users/sergey/.rvm/gems/ruby-2.5.5@some-gemset/bin/licensed:23:in `<top (required)>'

Even though

'{"type":"inspect","data":{"name":"ecc-jsbn","description":"ECC JS code based on JSBN","dist-tags":{"latest":"0.2.0"},"versions":["0.0.1","0.1.1","0.1.2","0.2.0"],"maintainers":[{"name":"quartzjer","email":"jeremie@jabber.org"},{"name":"rynomad","email":"nomad.ry@gmail.com"}],"time":{"modified":"2018-07-29T17:44:40.282Z","created":"2014-03-06T05:48:33.314Z","0.0.1":"2014-03-06T05:48:34.662Z","0.1.1":"2015-11-23T14:19:09.579Z","0.1.2":"2018-07-29T15:54:20.855Z","0.2.0":"2018-07-29T17:44:40.282Z"},"readmeFilename":"README.md","homepage":"https://github.com/quartzjer/ecc-jsbn","keywords":["jsbn","ecc","browserify"],"repository":{"type":"git","url":"git+https://github.com/quartzjer/ecc-jsbn.git"},"author":{"name":"Jeremie Miller","email":"jeremie@jabber.org","url":"http://jeremie.com/"},"bugs":{"url":"https://github.com/quartzjer/ecc-jsbn/issues"},"license":"MIT","users":{"mojaray2k":true},"version":"0.1.1","main":"index.js","dependencies":{"jsbn":"~0.1.0"},"gitHead":"d35a360352496721030da645e8054f07efc22487","scripts":{},"dist":{"shasum":"0fc73a9ed5f0d53c38193398523ef7e543777505","tarball":"https://registry.npmjs.org/ecc-jsbn/-/ecc-jsbn-0.1.1.tgz"},"directories":{}}}'

is parsed well when doing separately:

$ irb
2.5.5 :001 > require 'json'
 => true 
2.5.5 :002 > JSON.parse '{"type":"inspect","data":{"name":"ecc-jsbn","description":"ECC JS code based on JSBN","dist-tags":{"latest":"0.2.0"},"versions":["0.0.1","0.1.1","0.1.2","0.2.0"],"maintainers":[{"name":"quartzjer","email":"jeremie@jabber.org"},{"name":"rynomad","email":"nomad.ry@gmail.com"}],"time":{"modified":"2018-07-29T17:44:40.282Z","created":"2014-03-06T05:48:33.314Z","0.0.1":"2014-03-06T05:48:34.662Z","0.1.1":"2015-11-23T14:19:09.579Z","0.1.2":"2018-07-29T15:54:20.855Z","0.2.0":"2018-07-29T17:44:40.282Z"},"readmeFilename":"README.md","homepage":"https://github.com/quartzjer/ecc-jsbn","keywords":["jsbn","ecc","browserify"],"repository":{"type":"git","url":"git+https://github.com/quartzjer/ecc-jsbn.git"},"author":{"name":"Jeremie Miller","email":"jeremie@jabber.org","url":"http://jeremie.com/"},"bugs":{"url":"https://github.com/quartzjer/ecc-jsbn/issues"},"license":"MIT","users":{"mojaray2k":true},"version":"0.1.1","main":"index.js","dependencies":{"jsbn":"~0.1.0"},"gitHead":"d35a360352496721030da645e8054f07efc22487","scripts":{},"dist":{"shasum":"0fc73a9ed5f0d53c38193398523ef7e543777505","tarball":"https://registry.npmjs.org/ecc-jsbn/-/ecc-jsbn-0.1.1.tgz"},"directories":{}}}'
 => {"type"=>"inspect", "data"=>{"name"=>"ecc-jsbn", "description"=>"ECC JS code based on JSBN", "dist-tags"=>{"latest"=>"0.2.0"}, "versions"=>["0.0.1", "0.1.1", "0.1.2", "0.2.0"], "maintainers"=>[{"name"=>"quartzjer", "email"=>"jeremie@jabber.org"}, {"name"=>"rynomad", "email"=>"nomad.ry@gmail.com"}], "time"=>{"modified"=>"2018-07-29T17:44:40.282Z", "created"=>"2014-03-06T05:48:33.314Z", "0.0.1"=>"2014-03-06T05:48:34.662Z", "0.1.1"=>"2015-11-23T14:19:09.579Z", "0.1.2"=>"2018-07-29T15:54:20.855Z", "0.2.0"=>"2018-07-29T17:44:40.282Z"}, "readmeFilename"=>"README.md", "homepage"=>"https://github.com/quartzjer/ecc-jsbn", "keywords"=>["jsbn", "ecc", "browserify"], "repository"=>{"type"=>"git", "url"=>"git+https://github.com/quartzjer/ecc-jsbn.git"}, "author"=>{"name"=>"Jeremie Miller", "email"=>"jeremie@jabber.org", "url"=>"http://jeremie.com/"}, "bugs"=>{"url"=>"https://github.com/quartzjer/ecc-jsbn/issues"}, "license"=>"MIT", "users"=>{"mojaray2k"=>true}, "version"=>"0.1.1", "main"=>"index.js", "dependencies"=>{"jsbn"=>"~0.1.0"}, "gitHead"=>"d35a360352496721030da645e8054f07efc22487", "scripts"=>{}, "dist"=>{"shasum"=>"0fc73a9ed5f0d53c38193398523ef7e543777505", "tarball"=>"https://registry.npmjs.org/ecc-jsbn/-/ecc-jsbn-0.1.1.tgz"}, "directories"=>{}}} 
2.5.5 :003 > _["data"]
 => {"name"=>"ecc-jsbn", "description"=>"ECC JS code based on JSBN", "dist-tags"=>{"latest"=>"0.2.0"}, "versions"=>["0.0.1", "0.1.1", "0.1.2", "0.2.0"], "maintainers"=>[{"name"=>"quartzjer", "email"=>"jeremie@jabber.org"}, {"name"=>"rynomad", "email"=>"nomad.ry@gmail.com"}], "time"=>{"modified"=>"2018-07-29T17:44:40.282Z", "created"=>"2014-03-06T05:48:33.314Z", "0.0.1"=>"2014-03-06T05:48:34.662Z", "0.1.1"=>"2015-11-23T14:19:09.579Z", "0.1.2"=>"2018-07-29T15:54:20.855Z", "0.2.0"=>"2018-07-29T17:44:40.282Z"}, "readmeFilename"=>"README.md", "homepage"=>"https://github.com/quartzjer/ecc-jsbn", "keywords"=>["jsbn", "ecc", "browserify"], "repository"=>{"type"=>"git", "url"=>"git+https://github.com/quartzjer/ecc-jsbn.git"}, "author"=>{"name"=>"Jeremie Miller", "email"=>"jeremie@jabber.org", "url"=>"http://jeremie.com/"}, "bugs"=>{"url"=>"https://github.com/quartzjer/ecc-jsbn/issues"}, "license"=>"MIT", "users"=>{"mojaray2k"=>true}, "version"=>"0.1.1", "main"=>"index.js", "dependencies"=>{"jsbn"=>"~0.1.0"}, "gitHead"=>"d35a360352496721030da645e8054f07efc22487", "scripts"=>{}, "dist"=>{"shasum"=>"0fc73a9ed5f0d53c38193398523ef7e543777505", "tarball"=>"https://registry.npmjs.org/ecc-jsbn/-/ecc-jsbn-0.1.1.tgz"}, "directories"=>{}} 

Same with other JSONs, e.g.

JSON::ParserError: 765: unexpected token at '{"type":"inspect","data":{"name":"oauth-sign","description":"OAuth 1 signing. Formerly a vendor lib in mikeal/request, now a standalone module.","dist-tags":{"latest":"0.9.0"},"versions":["0.2.0","0.3.0","0.4.0","0.5.0","0.6.0","0.7.0","0.8.0","0.8.1","0.8.2","0.9.0"],"maintainers":[{"name":"mikeal","email":"mikeal.rogers@gmail.com"},{"name":"nylen","email":"jnylen@gmail.com"},{"name":"simov","email":"simeonvelichkov@gmail.com"}],"time":{"modified":"2018-08-02T18:04:01.221Z","created":"2013-03-01T20:23:20.098Z","0.2.0":"2013-03-01T20:23:20.779Z","0.3.0":"2013-04-22T05:12:37.269Z","0.4.0":"2013-09-23T22:58:47.722Z","0.5.0":"2014-10-13T15:50:26.714Z","0.6.0":"2015-01-04T01:30:55.552Z","0.7.0":"2015-03-21T20:51:35.145Z","0.8.0":"2015-05-28T10:43:18.008Z","0.8.1":"2016-01-31T07:18:27.473Z","0.8.2":"2016-05-04T21:13:21.752Z","0.9.0":"2018-08-02T18:03:51.602Z"},"author":{"name":"Mikeal Rogers","email":"mikeal.rogers@gmail.com","url":"http://www.futurealoof.com"},"repository":{"url":"git+https://github.com/mikeal/oauth-sign.git"},"homepage":"https://github.com/mikeal/oauth-sign#readme","bugs":{"url":"https://github.com/mikeal/oauth-sign/issues"},"readmeFilename":"README.md","license":"Apache-2.0","users":{"mojaray2k":true,"wheelo":true,"yorusi":true,"willwolffmyren":true},"version":"0.9.0","main":"index.js","files":["index.js"],"dependencies":{},"devDependencies":{},"optionalDependencies":{},"engines":{"node":"*"},"scripts":{"test":"node test.js"},"gitHead":"18a2513da6ba7a2c0cd8179170d7c296c7625137","dist":{"integrity":"sha512-fexhUFFPTGV8ybAtSIGbV6gOkSv8UtRbDBnAyLQw4QPKkgNlsH2ByPGtMUqdWkos6YCRmAqViwgZrJc/mRDzZQ==","shasum":"47a7b016baa68b5fa0ecf3dee08a85c679ac6455","tarball":"https://registry.npmjs.org/oauth-sign/-/oauth-sign-0.9.0.tgz","fileCount":4,"unpackedSize":13805,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJbY0eHCRA9TVsSAnZWagAAjUEQAIzrYuPpA5aQBQzEmivM\nmcseCarSrww/LRygkxzZjdoqlGOJtbWhIrZbwDhkRwe8jZ9tNxlikODsEpCN\nSHgNNlP4+vLRQTtzBbLRs9gNn+xFqKD5gLR6+6onSjjImykdqXmetAZqILFI\ncH66JIOBUkRFJcd31vMox78v/RJjeikg3rv/DLguuD5/7dvszzSfLEkA3aQr\nXq3PQUZoPFHzEhrv2FD2C4+9AsIL2BkSI57Gq19Q0HHGMvQ5qUIvRqlhDiDX\nVC1Ru74solr0yV2LKP/NSyCtL1AqBVq9Buzb85PRdZMcZ3Ir1ExflgfRon2h\n5Oxlo+ZfWe8cnCiMOOTxN6RBRDte8sK+somrg+9ZzTpK4/tGTHrNsm3dLze6\nQ3WVSizWpZJlgfmjD5RYrRvgNfKi+idAqgGL39W+1YNzQCvF7w9KFbhAMnn0\n4JEH/IrJlWHYoNP196ZQeonRLnJE4pl2IaBPEB7PGMCSm3HX0x291YnxI3ma\nyUxY12l09S9Xl54P8sFwa5UZsbvNxOICeEhzw7X8bGLCHqtXowlWCTo8qIUw\nAtRz84Iz8+6kXwxwf+SY+A365buiNHHJVD/AST6lmUQPbEQHGZUSERrZiHJj\n/xqtQEmQ9YjuWIDA34Py3Kl96wcAh8um8OISlZyZnBjx9ijiloBVucVM2pxX\n45tC\r\n=aokP\r\n-----END PGP SIGNATURE-----\r\n"},"directories":{}}}'

script/source-setup/yarn Show resolved Hide resolved
test/fixtures/yarn/package.json Show resolved Hide resolved
lib/licensed/sources/yarn.rb Show resolved Hide resolved
lib/licensed/sources/yarn.rb Show resolved Hide resolved
@jonabc
Copy link
Contributor Author

jonabc commented Dec 30, 2019

@krzysztof-pawlik-gat thanks for the feedback! I'll open a PR to address everything 🙇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a yarn dependency source
3 participants