filenames with spaces not always detected on mirrors

[poeml]

                                                                                                                                               [          ]

Issue migrated (2015-06-05) from old issue tracker http://mirrorbrain.org/issues/issue31

Title    filenames with spaces not always detected on mirrors
 Priority   bug            Status       resolved
Superseder                Nosy List     poeml, theuni
Assigned To poeml         Keywords

msg89 (view) Author: theuni Date: 2009-12-05.06:07:19

See here for reference:
http://mirrors.xbmc.org/addons/plugins/pictures/The%20Big%20Picture.tar.gz?mirrorlist

Some mirrors have the file, but aren't added to the list. Using that example the
following mirrors are 2 of the ones missing from that list:
http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/The%20Big%20Picture.tar.gz
http://mirror.netcologne.de/xbmc/addons/plugins/picture/The%20Big%20Picture.tar.gz

I haven't been able to track down why some work and others don't. But it's clear
that it's the whitespace that causes issues.

msg90 (view) Author: theuni Date: 2009-12-05.06:09:26

oops, typo in report.

http://mirror.netcologne.de/xbmc/addons/plugins/picture/The%20Big%20Picture.tar.gz
should be
http://mirror.netcologne.de/xbmc/addons/plugins/pictures/The%20Big%20Picture.tar.gz

msg91 (view) Author: poeml Date: 2009-12-05.11:57:42

I set up a little test instance here. The list of mirrors, and the URLs look fine, when I search in the database. Here's an example with one
file containing spaces, one not:

mirrorbrain@doozer:> mb file ls 'addons/plugins/pictures/iPhoto.tar.gz' -u
eu de 100 ok ok www.softliste.de http://www.softliste.de/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu de 100 ok ok mirror.netcologne.de http://mirror.netcologne.de/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu es 100 ok ok evorq.ugr.es http://evorq.ugr.es/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu fr 100 ok ok distrib-coffee.ipsl.jussieu.fr http://distrib-
coffee.ipsl.jussieu.fr/pub/mirrors/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu se 100 ok ok ftp.sunet.se http://ftp.sunet.se/pub/multimedia/xbmc/addons/plugins/pictures/iPhoto.tar.gz
na us 100 ok ok mirror.its.uidaho.edu http://mirror.its.uidaho.edu/pub/xbmc/addons/plugins/pictures/iPhoto.tar.gz
na us 100 ok ok www.gtlib.gatech.edu http://www.gtlib.gatech.edu/pub/xbmc/addons/plugins/pictures/iPhoto.tar.gz
na us 100 ok ok ftp.osuosl.org http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/iPhoto.tar.gz
mirrorbrain@doozer:> mb file ls 'addons/plugins/pictures/The Big Picture.tar.gz' -u
eu de 100 ok ok www.softliste.de http://www.softliste.de/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu de 100 ok ok mirror.netcologne.de http://mirror.netcologne.de/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu es 100 ok ok evorq.ugr.es http://evorq.ugr.es/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu fr 100 ok ok distrib-coffee.ipsl.jussieu.fr http://distrib-coffee.ipsl.jussieu.fr/pub/mirrors/xbmc/addons/plugins/pictures/The Big
Picture.tar.gz
eu se 100 ok ok ftp.sunet.se http://ftp.sunet.se/pub/multimedia/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok mirror.its.uidaho.edu http://mirror.its.uidaho.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok www.gtlib.gatech.edu http://www.gtlib.gatech.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok ftp.osuosl.org http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz

I also get all 8 mirrors in the mirror list:

http://xbmc.mirrorbrain.org/addons/plugins/pictures/iPhoto.tar.gz?mirrorlist
http://xbmc.mirrorbrain.org/addons/plugins/pictures/The%20Big%20Picture.tar.gz?mirrorlist

However, I just added the mirrors with their HTTP URL. Thus, I use HTTP for scanning, which seems to work. Maybe you have FTP and rsync URLs
configured - which would be preferred by the scanner.

So, I added the FTP URL for the netcologne mirror, and scan it again:

mirrorbrain@doozer:> mb scan netcolo
Sat Dec 5 12:55:51 2009 mirror.netcologne.de: starting
Sat Dec 5 12:55:51 2009 mirror.netcologne.de: total files before scan: 96
Sat Dec 5 12:55:51 2009 mirror.netcologne.de: no rsync, trying ftp
Sat Dec 5 12:55:52 2009 mirror.netcologne.de: scanned 69 files (60/s) in 1s
Sat Dec 5 12:55:52 2009 mirror.netcologne.de: files to be purged: 39
Sat Dec 5 12:55:52 2009 mirror.netcologne.de: total files after scan: 69
Sat Dec 5 12:55:52 2009 mirror.netcologne.de: purged old files in 0s.
Sat Dec 5 12:55:52 2009 mirror.netcologne.de: done.
Completed in 1 seconds
mirrorbrain@doozer:> mb file ls 'addons/plugins/pictures/The Big Picture.tar.gz' -u
eu de 100 ok ok www.softliste.de http://www.softliste.de/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu es 100 ok ok evorq.ugr.es http://evorq.ugr.es/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu fr 100 ok ok distrib-coffee.ipsl.jussieu.fr http://distrib-coffee.ipsl.jussieu.fr/pub/mirrors/xbmc/addons/plugins/pictures/The Big
Picture.tar.gz
eu se 100 ok ok ftp.sunet.se http://ftp.sunet.se/pub/multimedia/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok ftp.osuosl.org http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok www.gtlib.gatech.edu http://www.gtlib.gatech.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok mirror.its.uidaho.edu http://mirror.its.uidaho.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz

It's gone from the list. Thus, the problem is in the FTP scanner.

msg92 (view) Author: poeml Date: 2009-12-05.12:08:27

A verbose scan shows the following:

% mb scan netcolo -vvvvv --sql-debug
[...]

mirror.netcologne.de: ftp dir: addons/plugins/pictures
mirror.netcologne.de: -rw-r--r-- 1 804 804 39209 Nov 09 03:13 Phonebin.tar.gz
-rw-r--r-- 1 804 804 436921 Nov 09 03:13 Picasa.tar.gz
-rw-r--r-- 1 804 804 255436 Nov 09 03:13 The Big Picture.tar.gz
-rw-r--r-- 1 804 804 479633 Nov 09 03:12 flickr.tar.gz
-rw-r--r-- 1 804 804 2351538 Nov 09 03:12 iPhoto.tar.gz
-rw-r--r-- 1 804 804 21191 Nov 09 03:13 widelec.org.tar.gz
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/Phonebin.tar.gz
DELETE FROM temp1 WHERE id = 4
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/Picasa.tar.gz
DELETE FROM temp1 WHERE id = 5
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/flickr.tar.gz
DELETE FROM temp1 WHERE id = 7
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/iPhoto.tar.gz
DELETE FROM temp1 WHERE id = 8
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/widelec.org.tar.gz
DELETE FROM temp1 WHERE id = 9
mirror.netcologne.de: committing ftp dir addons/plugins/pictures

The file is seen via FTP (first half of the log), but no action taken when it comes to storing the file into the
database (second part of log). Staring at the scanner, line 638 now.

msg93 (view) Author: poeml Date: 2009-12-05.12:39:36

fixed in trunk:
http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/tools/scanner.pl?
r1=7905&r2=7904&pathrev=7905

msg94 (view) Author: poeml Date: 2009-12-05.16:10:36

I'm verifying the results that the new scanner brings by running it on all
OpenOffice.org mirrors that are scanned via FTP; it looks good.

msg95 (view) Author: poeml Date: 2009-12-05.21:21:25

fixed in 2.11.2. Thanks for the report!

msg101 (view) Author: theuni Date: 2009-12-09.07:44:41

Whoops, missed the reply.

Indeed this is fixed and working great. Thanks for the quick turnaround as usual.

History
         Date          User  Action              Args
2009-12-09 22:16:27 poeml  set    status: chatting -> resolved
2009-12-09 07:44:41 theuni set    status: resolved -> chatting
                                    messages: + msg101
2009-12-05 21:21:25 poeml  set    status: testing -> resolved
                                    messages: + msg95
2009-12-05 16:10:37 poeml  set    messages: + msg94
2009-12-05 12:39:36 poeml  set    status: in-progress -> testing
                                    messages: + msg93
2009-12-05 12:08:27 poeml  set    messages: + msg92
2009-12-05 11:57:43 poeml  set    messages: + msg91
2009-12-05 11:05:02 poeml  set    status: chatting -> in-progress
2009-12-05 06:09:26 theuni set    status: unread -> chatting
                                    messages: + msg90
2009-12-05 06:07:20 theuni create

(end of migrated issue)