withoutworldwriteables is a bad idea #1

timbunce opened this Issue Oct 11, 2010 · 11 comments


None yet

3 participants

The whole approach of -withoutworldwriteables seems wrong to me (and others I've discussed it with). It adds complication and confusion.

It also seems broken in that http://search.cpan.org/~pythian/DBD-Oracle-1.25/ shows both distro files, but wget -q -O - http://cpan.cpantesters.org/modules/02packages.details.txt.gz | zgrep 'DBD::Oracle ' only shows 0.24.

I believe it would be better to reject the tarball with message that explained the problem(s) and described how to fix them, including links to tools/utilities etc. (such as http://www.perlmonks.org/?node_id=731935) ?

Alternatively I'd support recognition of a class of "broken tarballs" and automatic in-place editing of permissions (like the script in http://www.perlmonks.org/?node_id=731935). Such tarballs are easy to recognize because the first entry is "drw-rw-rw-" which is clearly broken.


rafl commented Oct 11, 2010

FWIW, the entry for the toplevel directory doesn't need to be the first one in a tarball.

I also believe rejecting broken tarballs instead of attempting to repair them would be a good idea.

(Yes, it's reasonable to assume a broken tarball if there's a "drw-rw-rw-" entry anywhere. I agree that (helpful) rejection is better.)

The email that says "The distribution contains the following world writable ... not
being indexed" contains a link to http://use.perl.org/~bart/journal/38127 but the link is lost at the end of a long list of files and so is very unlikely to be seen.

More problems:

After uploading DBD-Oracle-1.26 I uploaded DBD-Oracle-1.25_brokentar.tar.gz (identical to the DBD-Oracle-1.25.tar.gz) as a test to see the indexer message.

I got an index report about world writable files (as mentioned above) but the new file, created by PAUSE was called "DBD-Oracle-1.25-withoutworldwriteables.tar.gz" not "DBD-Oracle-1.25_brokentar-withoutworldwriteables.tar.gz".

So now there are two copies of DBD-Oracle-1.25-withoutworldwriteables.tar.gz on CPAN. One in my directory and one in PYTHIAN.

andk commented Oct 12, 2010

If you send me both a notification and open an issue with the same text the discussion necessarily gets fragmented. I've answered the notification. I'll dig it out and report it here.

andk commented Oct 12, 2010

This was my answer to Tim in my Inbox folder. It is copied and pasted from over there but I inserted newlines for the listing so that it becomes readable.

(1) if the file was rejected by the indexer with a message that explained the problem(s) and described how to fix them, including
links to tools/utilities etc.?

The policy was for about two years that the file was rejected by the
indexer with a message that explained the problem(s) and described how
to fix them, including links to tools/utilities etc.

We even produced the -withoutworldwritables file in a place to
download from for quickest redeem. The result was that a lot of
distros on CPAN remained unindexed because the uploaders did not care.
FDavid Golden kept a list of these distros and when he reported they
were nearly hundred I knoew I had to change something. At that moment
we know that this one alternative approach does not work.

(2) you couldn't see world writables in 1.25

Then you must have looked with the wrong means

tar tvzf /home/ftp/pub/PAUSE/authors/id/P/PY/PYTHIAN/DBD-Oracle-1.25.tar.gz| head

drw-rw-rw- Administrator/0 0 2010-09-17 02:43 DBD-Oracle-1.25/

-rw-rw-rw- Administrator/0 60762 2010-09-17 02:01 DBD-Oracle-1.25/Changes

-rw-rw-rw- Administrator/0 129000 2010-09-17 01:37 DBD-Oracle-1.25/dbdimp.c

-rw-rw-rw- Administrator/0 15755 2010-09-17 01:37 DBD-Oracle-1.25/dbdimp.h

-rw-rw-rw- Administrator/0 2016 2010-09-17 01:37 DBD-Oracle-1.25/dbivport.h

drw-rw-rw- Administrator/0 0 2010-09-17 02:43 DBD-Oracle-1.25/hints/

-rw-rw-rw- Administrator/0 322 2010-05-28 21:29 DBD-Oracle-1.25/hints/dgux.pl

-rw-rw-rw- Administrator/0 198 2010-05-28 21:29 DBD-Oracle-1.25/hints/macos_bundle.syms

-rw-rw-rw- Administrator/0 23 2010-05-28 21:29 DBD-Oracle-1.25/hints/macos_lib.syms

-rw-rw-rw- Administrator/0 5076 2010-05-28 21:29 DBD-Oracle-1.25/hints/macos_syms.pl

All of them are world writable.

(3) wget -q -O - http://cpan.cpantesters.org/modules/02packages.details.txt.gz | zgrep 'DBD::Oracle '

I cannot reproduce that and if it was true, it most probably was a
temporary issue because the index lags behind the upload up to one
hour. In the meantime I see 1.26 there.

[Sorry for the duplication of the original report Andreas. I did mention I planned to report it elsewhere.]

(3) DBD-Oracle-1.25 was uploaded on Sept 17th. I was informed of the index problem, and confirmed it myself, about three weeks later. That seems like a bug somewhere. (The index did update within an hour or so of of my uploading DBD-Oracle-1.26.)

(2) Yes, a mistake on my part. (I'd unpacked the tar file and looked for world writable files in the output, but my umask had corrected them.) Sorry for the distraction.

(1) Why does http://search.cpan.org/~pythian/DBD-Oracle-1.25/ show both the -withoutworldwriteables and the original broken tar? That seems like a bug somewhere. From what you've said above, the original broken tar should not be visible.

(4) Why does http://search.cpan.org/dist/DBD-Oracle/ show DBD-Oracle-1.25-withoutworldwriteables even though 02packages.details.txt.gz indexes TIMB/DBD-Oracle-1.26.tar.gz? That's a bug somewhere (possibly caused by my uploading DBD-Oracle-1.25_brokentar.tar.gz as a test after uploading DBD-Oracle-1.26.tar.gz). Reindexing DBD-Oracle-1.26.tar.gz hasn't helped.

I appreciate that (1) and (4) may well be bugs in how search.cpan.org is tracking the index.

(5) A side issue: the code that generates the -withoutworldwriteables tarball doesn't fix directories to have (at least) user execute permission. So after downloading you can't browse or cd into the unpacked directories.

(6) I appreciate the desire to do something useful with 'broken' tarballs. My complaints mostly stem from apparent bugs being tickled by the current implementation of PAUSE and/or search.cpan.org. Having said that, I think a simpler approach would be to stop making worldwritables a special case. Correcting of world write and directory execute permissions could simply be standard practice applied to all uploads (optionally disabled by a META entry, in the rare case anyone would care).

andk commented Oct 13, 2010

Correcting of world write and directory execute permissions could
simply be standard practice applied to all uploads

I like this idea a lot. Yes, it's time to change the policy once
again, especially since so many incoming tarballs are so horribly
broken that removing the world writable bit does not help.

I've investigated a bit more and I think all bugs are now obvious to
me: the uploaded tarball PYTHIAN/DBD-Oracle-1.25.tar.gz was broken in
two ways. Directories had no x bit set and all files were world
writable. By correcting the world writable bit pause corrected only
one aspect. The broken directories were retained in the tarball
PYTHIAN/DBD-Oracle-1.25-withoutworldwriteables.tar.gz that pause
produced. That lead to the consequence that neither of the two
tarballs for 1.25 by pythian could be indexed and 1.24 remained in the
index file. That was the optimum pause could produce at that moment.
Pythian got two emails about the problem. I have forwarded them to you
in private mail.

I cannot blame search.cpan.org for displaying both (and later three)
broken tarballs. It's a way how we all are enabled to debug issues. On
the contrary, we need this level of visibility of bugs. And if we
protect the enduser from accidents we're getting the best out of it.

To summarize: none of the three broken 1.25 tarballs got indexed but
all three are visible and can be downloaded and people can use them.

The idea to routinely fix up more of the broken uploaded tarballs is
well received and goes on my todo list. I'll most probably stick with
the policy to retain the broken stuff on CPAN, but I'll invent a
different format for that. Something like a timestamped pair of
original tarball and a report to increase visibility and

I'm glad you like the idea of applying tarball cleanup as standard practice. (And thanks for the emails.)

However, having http://search.cpan.org/dist/DBD-Oracle/ (with no version number) show 1.25 when the index shows 1.24 is clearly a bug and very misleading to users. Is there way to report bugs in search.cpan.org (other than emailing cpansearch @perl.org)?

andk commented Oct 13, 2010

My experience with cpansearch@perl.org is optimal and I have no better address

andk commented Feb 5, 2012

96f56ec has now the rewrite code in the daemon.

@andk andk closed this Feb 5, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment