New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use MetaCPAN API query data as fallback for package metadata #849

Merged
merged 9 commits into from Nov 7, 2015

Conversation

Projects
None yet
2 participants
@tomeon

tomeon commented Feb 10, 2015

Some CPAN packages lack META/MYMETA files, which causes fpm to raise an error and exit. However, the MetaCPAN query returns distribution metadata, so there's no need for the lack of a META file to cause a fatal error.

This pull request merges the MetaCPAN query data with whatever data can be read from a local META file, with the local data taking precedence. If the metadata hash is empty after the merge, a fatal error is raised.

Matt Schreiber added some commits Feb 10, 2015

@tomeon

This comment has been minimized.

tomeon commented Feb 11, 2015

On deeper exploration, it appears that some of the attributes that should appear in the metadata hash are not included in the results of the MetaCPAN query. For instance, the 'license' field is associated with queries to http://api.metacpan.org/v0/release, but the code currently queries only http://api.metacpan.org/v0/module.

There are a couple of options for getting additional data:

  • More GET requests to the MetaCPAN endpoints, using data gleaned from earlier requests to build up a fuller picture of a package's metadata; or
  • Integrating the elasticsearch gem and using that to query MetaCPAN.

MetaCPAN's only rule is 'be polite', which seems to be operationalized as 'If you're going to query more than 5000 modules, use Elasticsearch'. I don't know what the likelihood is that fpm users will be making such large queries, so I'd tend to go with the first, simpler approach. Thoughts?

Matt Schreiber Matt Schreiber
Merge https://github.com/jordansissel/fpm
* https://github.com/jordansissel/fpm: (29 commits)
  Cleaned up the p5p code and revised work-flow
  rpm v5 changes the name of the file digest macro
  Bump arr-pm version for a small bugfix in RPM::File#config_files
  Add RPM input tests
  Remove debugging output when using rpm triggers
  change deprecated 'File.exists?' to 'File.exist?'
  Strip trailing whitespace
  update name in CONTRIBUTORS
  Add rubocop config.
  Continuation of efforts to improve how fpm uses rspec.
  Add support for Solaris 11 style IPS packages using *.p5p format
  rspec: fix lintian test
  Grand refactor to use my newer rspec style
  Fix a failing test on OSX that required 'dpkg-deb'
  Fix for *debian* packages for #875
  #875 Fix cases where upgrade or install scripts are specified, but not both
  PR 853: --[no-]cpan-test now works as advertised
  Add improved abort-install and abort-remove
  Nothing can be found in PATH if its not set
  Until python wheel format is supported, tell pip to --no-use-wheel. See #587.
  ...

@jordansissel jordansissel force-pushed the jordansissel:master branch 2 times, most recently from 715ab62 to 9866c6d Apr 24, 2015

@jordansissel

This comment has been minimized.

Owner

jordansissel commented May 7, 2015

Reviewed the code - looks fine to me :)

As for your other concern for fetching additional data, I am happy to have that as well. If you wish this patch merged first and want to work on the extra data fetching later, I'm OK with that. Let me know :)

Matt Schreiber added some commits Jun 4, 2015

Matt Schreiber Matt Schreiber
Merge https://github.com/jordansissel/fpm
* https://github.com/jordansissel/fpm: (21 commits)
  make sandboxing non-core modules optional
  Remove python_install_bin setting; this was causing the tests to fail on OSX
  Whitespace
  Revert unintentional change
  Remove unnecessary requires lines from  spec
  Add empty as output target for testing purposes
  Add before-install script to sh package type
  Add  to pre- and post- removal scripts to make them safer
  fixed typo in command.rb
  fix exclude-file to exclude_file issue
  For Debian packages, register files in `/etc` as conf files
  List loaded package types on the help screen
  Fixed github link title in RAEDME
  First whack at translating deb scripts/conffiles into rpm
  Add tests for python dependency manipulation. Also switch spec/fixtures/python/setup.py from distutils to setuptools.
  Add --python-disable-dependency option to disable specific python dependencies.
  Add command line flag for --exclude-file.
  Add commandline flag for --exclude-file to support a file listing exclude patterns.
  Add some basic specs for the SH package
  Add package metadata to slug file format
  ...
Matt Schreiber Matt Schreiber
Merge branch 'master' of https://github.com/jordansissel/fpm
* 'master' of https://github.com/jordansissel/fpm:
  Add tests to cover empty and one-line descriptions for rpms
  Don't crash on empty description when building RPM
  Fix Find.find usage to work under Ruby 1.8.x
  Fix minor typo
  mark release date
  Version bump
  Update changelog based on changes from v1.3.3 to master
  typo: installe -> installed
  Ensure file permission of control/shlibs
  Update deb_spec.rb
  Update deb.rb
  Remove virtualenv debug code
  Some more virtualenv polish
  Move copy_metadata so it can be used in more places
  Add virtualenv to the party
  Scope --rpm-dist to rpm.rb
  Support RPM dist tag
Matt Schreiber Matt Schreiber
initial commit of new rspec test
of CPAN module without META.* file
@jordansissel

This comment has been minimized.

Owner

jordansissel commented Nov 7, 2015

Code looks ok; tests pass.

Merging!

jordansissel added a commit that referenced this pull request Nov 7, 2015

Merge pull request #849 from BaxterStockman/cpan-metadata
Use MetaCPAN API query data as fallback for package metadata

@jordansissel jordansissel merged commit c1eaae4 into jordansissel:master Nov 7, 2015

jordansissel added a commit that referenced this pull request Jun 20, 2016

Merge pull request #849 from BaxterStockman/cpan-metadata
Use MetaCPAN API query data as fallback for package metadata
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment