New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download_url QA. (was: "module" search is too restrictive?) #483

Closed
karenetheridge opened this Issue May 19, 2016 · 21 comments

Comments

Projects
8 participants
@karenetheridge

karenetheridge commented May 19, 2016

from #metacpan on May 16:

16:00 < ether> I'm not sure if this is a perlbrew or metacpan issue -- are the search args being passed here correct? miyagawa/cpanminus#501
16:01 < ether> if 'path' => 'module' is wrong, what should it be instead?

@oalders

This comment has been minimized.

Show comment
Hide comment
@oalders

oalders May 19, 2016

Member

Sadly, I don't think there's a quick fix here. Having said that, v1 does have an endpoint which was built specifically for cpanm: https://github.com/CPAN-API/cpan-api/blob/mi/es2/lib/MetaCPAN/Document/File/Set.pm#L110

I think it still needs some testing, but this was done by @clintongormley to deal specifically with the needs of that client.

Member

oalders commented May 19, 2016

Sadly, I don't think there's a quick fix here. Having said that, v1 does have an endpoint which was built specifically for cpanm: https://github.com/CPAN-API/cpan-api/blob/mi/es2/lib/MetaCPAN/Document/File/Set.pm#L110

I think it still needs some testing, but this was done by @clintongormley to deal specifically with the needs of that client.

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley May 20, 2016

Member

Yeah I couldn't figure out how to build a version of metacpan containing the docs I needed for testing, but that API seemed to do the right thing if I remember correctly.

@miyagawa may be able to provide the relevant test cases.

Importantly, the find_download_url is much more efficient than a number of other metacpan queries because it uses nested queries to decide which docs to return, while in other places in metacpan, all possible matching docs are returned then whittled down with perl code.

@karenetheridge I don't really understand what this means

The wrong module/distribution is found when searching in --dev mode, when a .pod module supercedes a .pm.

but if we could understand the logic that needs to be applied (based on the fields available in metacpan) then possibly we could rejig the query to make it work

Member

clintongormley commented May 20, 2016

Yeah I couldn't figure out how to build a version of metacpan containing the docs I needed for testing, but that API seemed to do the right thing if I remember correctly.

@miyagawa may be able to provide the relevant test cases.

Importantly, the find_download_url is much more efficient than a number of other metacpan queries because it uses nested queries to decide which docs to return, while in other places in metacpan, all possible matching docs are returned then whittled down with perl code.

@karenetheridge I don't really understand what this means

The wrong module/distribution is found when searching in --dev mode, when a .pod module supercedes a .pm.

but if we could understand the logic that needs to be applied (based on the fields available in metacpan) then possibly we could rejig the query to make it work

@ranguard

This comment has been minimized.

Show comment
Hide comment
@ranguard

ranguard May 20, 2016

Member

V1 (under heavy development, which includes us re-indexing everything sometime) is available at: https://api-v1.metacpan.org/ e.g. https://api-v1.metacpan.org/author/ - if anyone wants to play

Member

ranguard commented May 20, 2016

V1 (under heavy development, which includes us re-indexing everything sometime) is available at: https://api-v1.metacpan.org/ e.g. https://api-v1.metacpan.org/author/ - if anyone wants to play

@karenetheridge

This comment has been minimized.

Show comment
Hide comment
@karenetheridge

karenetheridge May 20, 2016

@karenetheridge I don't really understand what this means

The wrong module/distribution is found when searching in --dev mode, when a .pod module supercedes a .pm.

What's happening here is:

  • cpanm --dev is doing a module search for Moose::Cookbook::Legacy::Debugging_BaseClassReplacement, and gets a result of DOY/Moose-2.0604.tar.gz
  • it is actually in the 02packages index pointing to the most recent Moose release, ETHER/2.1801! so that is the distribution that should be returned.
  • 2.0604 is the last Moose release where that module was a .pm file; in subsequent releases it became a .pod file -- therefore the search is too restrictive; it should be as inclusive as the PAUSE index itself is (which respects .pod files)

karenetheridge commented May 20, 2016

@karenetheridge I don't really understand what this means

The wrong module/distribution is found when searching in --dev mode, when a .pod module supercedes a .pm.

What's happening here is:

  • cpanm --dev is doing a module search for Moose::Cookbook::Legacy::Debugging_BaseClassReplacement, and gets a result of DOY/Moose-2.0604.tar.gz
  • it is actually in the 02packages index pointing to the most recent Moose release, ETHER/2.1801! so that is the distribution that should be returned.
  • 2.0604 is the last Moose release where that module was a .pm file; in subsequent releases it became a .pod file -- therefore the search is too restrictive; it should be as inclusive as the PAUSE index itself is (which respects .pod files)
@karenetheridge

This comment has been minimized.

Show comment
Hide comment
@karenetheridge

karenetheridge May 24, 2016

Here is another example of a similar query failure (the wrong distribution is found for the module 'encoding'):

$ cpanm --dev encoding
--> Working on encoding
Fetching http://mirrors.gossamer-threads.com/CPAN/authors/id/D/DA/DANKOGAI/Encode-2.48.tar.gz ... OK
Configuring Encode-2.48 ... OK
Building and testing Encode-2.48 ... ^C

$ grep encoding ~/.cpanm/02packages.details.txt
encoding                           2.17  D/DA/DANKOGAI/Encode-2.84.tar.gz

karenetheridge commented May 24, 2016

Here is another example of a similar query failure (the wrong distribution is found for the module 'encoding'):

$ cpanm --dev encoding
--> Working on encoding
Fetching http://mirrors.gossamer-threads.com/CPAN/authors/id/D/DA/DANKOGAI/Encode-2.48.tar.gz ... OK
Configuring Encode-2.48 ... OK
Building and testing Encode-2.48 ... ^C

$ grep encoding ~/.cpanm/02packages.details.txt
encoding                           2.17  D/DA/DANKOGAI/Encode-2.84.tar.gz
@karenetheridge

This comment has been minimized.

Show comment
Hide comment
@karenetheridge

karenetheridge May 24, 2016

Here's another bizarre bad result:

$ cpanm --dev Module::Install::Base::FakeAdmin
--> Working on Module::Install::Base::FakeAdmin
Fetching http://www.cpan.org/authors/id/A/AU/AUTRIJUS/Language-Haskell-0.01.tar.gz ... OK
Configuring Language-Haskell-0.01 ... OK
Building and testing Language-Haskell-0.01 ... FAIL

Module::Install::Base::FakeAdmin is indexed to the Module-Install distribution. Why is it coming back with AUTRIJUS/Language-Haskell-0.01.tar.gz ?!

karenetheridge commented May 24, 2016

Here's another bizarre bad result:

$ cpanm --dev Module::Install::Base::FakeAdmin
--> Working on Module::Install::Base::FakeAdmin
Fetching http://www.cpan.org/authors/id/A/AU/AUTRIJUS/Language-Haskell-0.01.tar.gz ... OK
Configuring Language-Haskell-0.01 ... OK
Building and testing Language-Haskell-0.01 ... FAIL

Module::Install::Base::FakeAdmin is indexed to the Module-Install distribution. Why is it coming back with AUTRIJUS/Language-Haskell-0.01.tar.gz ?!

karenetheridge added a commit to karenetheridge/misc that referenced this issue Jun 14, 2016

stop installing so many modules with --dev
metacpan search returns incorrect results for some files, causing older
distributions to be installed and shadowing the newer version

see metacpan/metacpan-api#483
@karenetheridge

This comment has been minimized.

Show comment
Hide comment
@karenetheridge

karenetheridge Jun 15, 2016

woohoo, https://api-v1.metacpan.org/module/Moose::Cookbook::Legacy::Debugging_BaseClassReplacement

returns:

{
...
   "path" : "lib/Moose/Cookbook/Legacy/Debugging_BaseClassReplacement.pod",
   "download_url" : "https://cpan.metacpan.org/authors/id/E/ET/ETHER/Moose-2.1804.tar.gz",
...
}

karenetheridge commented Jun 15, 2016

woohoo, https://api-v1.metacpan.org/module/Moose::Cookbook::Legacy::Debugging_BaseClassReplacement

returns:

{
...
   "path" : "lib/Moose/Cookbook/Legacy/Debugging_BaseClassReplacement.pod",
   "download_url" : "https://cpan.metacpan.org/authors/id/E/ET/ETHER/Moose-2.1804.tar.gz",
...
}
@karenetheridge

This comment has been minimized.

Show comment
Hide comment
@karenetheridge

karenetheridge Jun 15, 2016

However, this is still a bad result:

https://api-v1.metacpan.org/module/Module::Install::Base::FakeAdmin

{
   "download_url" : "https://cpan.metacpan.org/authors/id/A/AU/AUTRIJUS/Language-Haskell-0.01.tar.gz",
...
}

karenetheridge commented Jun 15, 2016

However, this is still a bad result:

https://api-v1.metacpan.org/module/Module::Install::Base::FakeAdmin

{
   "download_url" : "https://cpan.metacpan.org/authors/id/A/AU/AUTRIJUS/Language-Haskell-0.01.tar.gz",
...
}
@oalders

This comment has been minimized.

Show comment
Hide comment
@oalders

oalders Jun 15, 2016

Member

@karenetheridge This is @clintongormley's new endpoint. We should include that in this discussion.

https://api-v1.metacpan.org/download_url/Module::Install::Base::FakeAdmin

{
   "date" : "2015-05-02T01:06:38",
   "status" : "latest",
   "version" : "0",
   "download_url" : "https://cpan.metacpan.org/authors/id/E/ET/ETHER/Module-Install-1.16.tar.gz"
}

https://api-v1.metacpan.org/download_url/Moose::Cookbook::Legacy::Debugging_BaseClassReplacement

{
   "download_url" : "https://cpan.metacpan.org/authors/id/E/ET/ETHER/Moose-2.0802.tar.gz",
   "version" : "2.0802",
   "status" : "cpan",
   "date" : "2013-05-07T17:10:09"
}
Member

oalders commented Jun 15, 2016

@karenetheridge This is @clintongormley's new endpoint. We should include that in this discussion.

https://api-v1.metacpan.org/download_url/Module::Install::Base::FakeAdmin

{
   "date" : "2015-05-02T01:06:38",
   "status" : "latest",
   "version" : "0",
   "download_url" : "https://cpan.metacpan.org/authors/id/E/ET/ETHER/Module-Install-1.16.tar.gz"
}

https://api-v1.metacpan.org/download_url/Moose::Cookbook::Legacy::Debugging_BaseClassReplacement

{
   "download_url" : "https://cpan.metacpan.org/authors/id/E/ET/ETHER/Moose-2.0802.tar.gz",
   "version" : "2.0802",
   "status" : "cpan",
   "date" : "2013-05-07T17:10:09"
}
@karenetheridge

This comment has been minimized.

Show comment
Hide comment
@karenetheridge

karenetheridge Jun 15, 2016

@oalders bizarre! of those two searches, now Module::Install::Base::FakeAdmin is correct and Moose::Cookbook::Legacy::Debugging_BaseClassReplacement is wrong :)

karenetheridge commented Jun 15, 2016

@oalders bizarre! of those two searches, now Module::Install::Base::FakeAdmin is correct and Moose::Cookbook::Legacy::Debugging_BaseClassReplacement is wrong :)

@oalders

This comment has been minimized.

Show comment
Hide comment
@oalders

oalders Jun 15, 2016

Member

:)

We should make sure this is correct before migrating officially to v1. For anyone who wants to hack on this, the relevant code is at https://github.com/CPAN-API/cpan-api/blob/master/lib/MetaCPAN/Document/File/Set.pm#L147

Member

oalders commented Jun 15, 2016

:)

We should make sure this is correct before migrating officially to v1. For anyone who wants to hack on this, the relevant code is at https://github.com/CPAN-API/cpan-api/blob/master/lib/MetaCPAN/Document/File/Set.pm#L147

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley Jun 16, 2016

Member

@oalders Sorry I don't have the time to delve into this in detail, but if you need any help with specific Elasticsearch queries (eg I need to change the logic to do X) I'm happy to help.

Member

clintongormley commented Jun 16, 2016

@oalders Sorry I don't have the time to delve into this in detail, but if you need any help with specific Elasticsearch queries (eg I need to change the logic to do X) I'm happy to help.

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley Jun 16, 2016

Member

btw the blocker for me with this endpoint was that I couldn't figure out how to write the tests

Member

clintongormley commented Jun 16, 2016

btw the blocker for me with this endpoint was that I couldn't figure out how to write the tests

@skaji

This comment has been minimized.

Show comment
Hide comment
@skaji

skaji Jun 19, 2016

Contributor

(From IRC...)
I'm currently trying to move over cpanminus MetaCPAN client to v1.

https://api-v1.metacpan.org/download_url/Net::LDAP::Server::Test?version=%3D%3D%200.14
(here the version query is "== 0.14") returns "status cpan" but I think it should be "status backpan".

In fact,
http://www.cpan.org/authors/id/K/KA/KARMAN/Net-LDAP-Server-Test-0.14.tar.gz returns 404 Not Found
http://backpan.perl.org/authors/id/K/KA/KARMAN/Net-LDAP-Server-Test-0.14.tar.gz returns 200 OK

Note: cpanminus has a test for backpan search
https://github.com/miyagawa/cpanminus/blob/menlo/xt/cli/backpan_search.t#L5-L8

Contributor

skaji commented Jun 19, 2016

(From IRC...)
I'm currently trying to move over cpanminus MetaCPAN client to v1.

https://api-v1.metacpan.org/download_url/Net::LDAP::Server::Test?version=%3D%3D%200.14
(here the version query is "== 0.14") returns "status cpan" but I think it should be "status backpan".

In fact,
http://www.cpan.org/authors/id/K/KA/KARMAN/Net-LDAP-Server-Test-0.14.tar.gz returns 404 Not Found
http://backpan.perl.org/authors/id/K/KA/KARMAN/Net-LDAP-Server-Test-0.14.tar.gz returns 200 OK

Note: cpanminus has a test for backpan search
https://github.com/miyagawa/cpanminus/blob/menlo/xt/cli/backpan_search.t#L5-L8

@skaji

This comment has been minimized.

Show comment
Hide comment
@skaji

skaji Jun 19, 2016

Contributor

Another issue related backpan:
backpan should be excluded when searching dev releases.
https://github.com/CPAN-API/cpan-api/blob/master/lib/MetaCPAN/Document/File/Set.pm#L158
https://github.com/miyagawa/cpanminus/blob/menlo/lib/Menlo/Index/MetaCPAN.pm#L213-L214

But actually https://api-v1.metacpan.org/download_url/Test::Simple?dev=1 returns

{
   "date" : "2011-02-28T13:00:05",
   "version" : "2.00_07",
   "download_url" : "https://cpan.metacpan.org/authors/id/M/MS/MSCHWERN/Test-Simple-2.00_07.tar.gz",
   "status" : "cpan"
}

here 2.00_07 is in backpan.

Please let me know when I should open a new issue.

Contributor

skaji commented Jun 19, 2016

Another issue related backpan:
backpan should be excluded when searching dev releases.
https://github.com/CPAN-API/cpan-api/blob/master/lib/MetaCPAN/Document/File/Set.pm#L158
https://github.com/miyagawa/cpanminus/blob/menlo/lib/Menlo/Index/MetaCPAN.pm#L213-L214

But actually https://api-v1.metacpan.org/download_url/Test::Simple?dev=1 returns

{
   "date" : "2011-02-28T13:00:05",
   "version" : "2.00_07",
   "download_url" : "https://cpan.metacpan.org/authors/id/M/MS/MSCHWERN/Test-Simple-2.00_07.tar.gz",
   "status" : "cpan"
}

here 2.00_07 is in backpan.

Please let me know when I should open a new issue.

@skaji

This comment has been minimized.

Show comment
Hide comment
@skaji
Contributor

skaji commented Jul 6, 2016

@mickeyn

This comment has been minimized.

Show comment
Hide comment
@mickeyn

mickeyn Jul 6, 2016

Contributor

@skaji I was doing some status cleanups/updates during the weekend, they were a bit off compared to v0. hope they are good now.

Contributor

mickeyn commented Jul 6, 2016

@skaji I was doing some status cleanups/updates during the weekend, they were a bit off compared to v0. hope they are good now.

@skaji

This comment has been minimized.

Show comment
Hide comment
@skaji

skaji Jul 6, 2016

Contributor

Let's go back to @karenetheridge's original issue.

(a) encoding -> solved

  • D/DA/DANKOGAI/Encode-2.84.tar.gz has encoding 2.17 (== 2.170)
  • D/DA/DANKOGAI/Encode-2.48.tar.gz has encoding 2.6_01 (== 2.601)

So it is correct that dev search of MetaCPAN v0 (and cpanm --dev encoding) returns Encode-2.48.

(b) Module::Install::Base::FakeAdmin -> solved

Now MetaCPAN v1 returns Module-Install. http://fastapi.metacpan.org/v1/download_url/Module::Install::Base::FakeAdmin?dev=1
http://fastapi.metacpan.org/v1/download_url/Module::Install::Base::FakeAdmin

(c) Moose::Cookbook::Legacy::Debugging_BaseClassReplacement (=MCLD) -> NOT SOLVED

So It seems that MetaCPAN does not count META.json correctly.
We could refer to Parse::LocalDistribution for how PAUSE parses distributions.

Contributor

skaji commented Jul 6, 2016

Let's go back to @karenetheridge's original issue.

(a) encoding -> solved

  • D/DA/DANKOGAI/Encode-2.84.tar.gz has encoding 2.17 (== 2.170)
  • D/DA/DANKOGAI/Encode-2.48.tar.gz has encoding 2.6_01 (== 2.601)

So it is correct that dev search of MetaCPAN v0 (and cpanm --dev encoding) returns Encode-2.48.

(b) Module::Install::Base::FakeAdmin -> solved

Now MetaCPAN v1 returns Module-Install. http://fastapi.metacpan.org/v1/download_url/Module::Install::Base::FakeAdmin?dev=1
http://fastapi.metacpan.org/v1/download_url/Module::Install::Base::FakeAdmin

(c) Moose::Cookbook::Legacy::Debugging_BaseClassReplacement (=MCLD) -> NOT SOLVED

So It seems that MetaCPAN does not count META.json correctly.
We could refer to Parse::LocalDistribution for how PAUSE parses distributions.

@skaji

This comment has been minimized.

Show comment
Hide comment
@skaji

skaji Aug 30, 2016

Contributor

(deleted)

Contributor

skaji commented Aug 30, 2016

(deleted)

@oalders oalders changed the title from "module" search is too restrictive? to download_url QA. (was: "module" search is too restrictive?) Nov 17, 2016

@haarg

This comment has been minimized.

Show comment
Hide comment
@haarg

haarg Nov 17, 2016

Member

The issues with Moose's pod files is #336

Member

haarg commented Nov 17, 2016

The issues with Moose's pod files is #336

@oalders

This comment has been minimized.

Show comment
Hide comment
@oalders

oalders Nov 17, 2016

Member

This looks good on our end. @skaji could you open a new ticket if you find anything else?

Member

oalders commented Nov 17, 2016

This looks good on our end. @skaji could you open a new ticket if you find anything else?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment