I set up a little test instance here. The list of mirrors, and the URLs look fine, when I search in the database. Here's an example with one
file containing spaces, one not:
However, I just added the mirrors with their HTTP URL. Thus, I use HTTP for scanning, which seems to work. Maybe you have FTP and rsync URLs
configured - which would be preferred by the scanner.
So, I added the FTP URL for the netcologne mirror, and scan it again:
mirror.netcologne.de: ftp dir: addons/plugins/pictures
mirror.netcologne.de: -rw-r--r-- 1 804 804 39209 Nov 09 03:13 Phonebin.tar.gz
-rw-r--r-- 1 804 804 436921 Nov 09 03:13 Picasa.tar.gz
-rw-r--r-- 1 804 804 255436 Nov 09 03:13 The Big Picture.tar.gz
-rw-r--r-- 1 804 804 479633 Nov 09 03:12 flickr.tar.gz
-rw-r--r-- 1 804 804 2351538 Nov 09 03:12 iPhoto.tar.gz
-rw-r--r-- 1 804 804 21191 Nov 09 03:13 widelec.org.tar.gz
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/Phonebin.tar.gz
DELETE FROM temp1 WHERE id = 4
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/Picasa.tar.gz
DELETE FROM temp1 WHERE id = 5
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/flickr.tar.gz
DELETE FROM temp1 WHERE id = 7
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/iPhoto.tar.gz
DELETE FROM temp1 WHERE id = 8
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/widelec.org.tar.gz
DELETE FROM temp1 WHERE id = 9
mirror.netcologne.de: committing ftp dir addons/plugins/pictures
The file is seen via FTP (first half of the log), but no action taken when it comes to storing the file into the
database (second part of log). Staring at the scanner, line 638 now.
Issue migrated (2015-06-05) from old issue tracker http://mirrorbrain.org/issues/issue31
msg89 (view) Author: theuni Date: 2009-12-05.06:07:19
See here for reference:
http://mirrors.xbmc.org/addons/plugins/pictures/The%20Big%20Picture.tar.gz?mirrorlist
Some mirrors have the file, but aren't added to the list. Using that example the
following mirrors are 2 of the ones missing from that list:
http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/The%20Big%20Picture.tar.gz
http://mirror.netcologne.de/xbmc/addons/plugins/picture/The%20Big%20Picture.tar.gz
I haven't been able to track down why some work and others don't. But it's clear
that it's the whitespace that causes issues.
msg90 (view) Author: theuni Date: 2009-12-05.06:09:26
oops, typo in report.
http://mirror.netcologne.de/xbmc/addons/plugins/picture/The%20Big%20Picture.tar.gz
should be
http://mirror.netcologne.de/xbmc/addons/plugins/pictures/The%20Big%20Picture.tar.gz
msg91 (view) Author: poeml Date: 2009-12-05.11:57:42
I set up a little test instance here. The list of mirrors, and the URLs look fine, when I search in the database. Here's an example with one
file containing spaces, one not:
mirrorbrain@doozer:
> mb file ls 'addons/plugins/pictures/iPhoto.tar.gz' -u> mb file ls 'addons/plugins/pictures/The Big Picture.tar.gz' -ueu de 100 ok ok www.softliste.de http://www.softliste.de/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu de 100 ok ok mirror.netcologne.de http://mirror.netcologne.de/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu es 100 ok ok evorq.ugr.es http://evorq.ugr.es/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu fr 100 ok ok distrib-coffee.ipsl.jussieu.fr http://distrib-
coffee.ipsl.jussieu.fr/pub/mirrors/xbmc/addons/plugins/pictures/iPhoto.tar.gz
eu se 100 ok ok ftp.sunet.se http://ftp.sunet.se/pub/multimedia/xbmc/addons/plugins/pictures/iPhoto.tar.gz
na us 100 ok ok mirror.its.uidaho.edu http://mirror.its.uidaho.edu/pub/xbmc/addons/plugins/pictures/iPhoto.tar.gz
na us 100 ok ok www.gtlib.gatech.edu http://www.gtlib.gatech.edu/pub/xbmc/addons/plugins/pictures/iPhoto.tar.gz
na us 100 ok ok ftp.osuosl.org http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/iPhoto.tar.gz
mirrorbrain@doozer:
eu de 100 ok ok www.softliste.de http://www.softliste.de/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu de 100 ok ok mirror.netcologne.de http://mirror.netcologne.de/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu es 100 ok ok evorq.ugr.es http://evorq.ugr.es/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu fr 100 ok ok distrib-coffee.ipsl.jussieu.fr http://distrib-coffee.ipsl.jussieu.fr/pub/mirrors/xbmc/addons/plugins/pictures/The Big
Picture.tar.gz
eu se 100 ok ok ftp.sunet.se http://ftp.sunet.se/pub/multimedia/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok mirror.its.uidaho.edu http://mirror.its.uidaho.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok www.gtlib.gatech.edu http://www.gtlib.gatech.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok ftp.osuosl.org http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
I also get all 8 mirrors in the mirror list:
http://xbmc.mirrorbrain.org/addons/plugins/pictures/iPhoto.tar.gz?mirrorlist
http://xbmc.mirrorbrain.org/addons/plugins/pictures/The%20Big%20Picture.tar.gz?mirrorlist
However, I just added the mirrors with their HTTP URL. Thus, I use HTTP for scanning, which seems to work. Maybe you have FTP and rsync URLs
configured - which would be preferred by the scanner.
So, I added the FTP URL for the netcologne mirror, and scan it again:
mirrorbrain@doozer:
> mb scan netcolo> mb file ls 'addons/plugins/pictures/The Big Picture.tar.gz' -uSat Dec 5 12:55:51 2009 mirror.netcologne.de: starting
Sat Dec 5 12:55:51 2009 mirror.netcologne.de: total files before scan: 96
Sat Dec 5 12:55:51 2009 mirror.netcologne.de: no rsync, trying ftp
Sat Dec 5 12:55:52 2009 mirror.netcologne.de: scanned 69 files (60/s) in 1s
Sat Dec 5 12:55:52 2009 mirror.netcologne.de: files to be purged: 39
Sat Dec 5 12:55:52 2009 mirror.netcologne.de: total files after scan: 69
Sat Dec 5 12:55:52 2009 mirror.netcologne.de: purged old files in 0s.
Sat Dec 5 12:55:52 2009 mirror.netcologne.de: done.
Completed in 1 seconds
mirrorbrain@doozer:
eu de 100 ok ok www.softliste.de http://www.softliste.de/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu es 100 ok ok evorq.ugr.es http://evorq.ugr.es/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
eu fr 100 ok ok distrib-coffee.ipsl.jussieu.fr http://distrib-coffee.ipsl.jussieu.fr/pub/mirrors/xbmc/addons/plugins/pictures/The Big
Picture.tar.gz
eu se 100 ok ok ftp.sunet.se http://ftp.sunet.se/pub/multimedia/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok ftp.osuosl.org http://ftp.osuosl.org/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok www.gtlib.gatech.edu http://www.gtlib.gatech.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
na us 100 ok ok mirror.its.uidaho.edu http://mirror.its.uidaho.edu/pub/xbmc/addons/plugins/pictures/The Big Picture.tar.gz
It's gone from the list. Thus, the problem is in the FTP scanner.
msg92 (view) Author: poeml Date: 2009-12-05.12:08:27
A verbose scan shows the following:
% mb scan netcolo -vvvvv --sql-debug
[...]
mirror.netcologne.de: ftp dir: addons/plugins/pictures
mirror.netcologne.de: -rw-r--r-- 1 804 804 39209 Nov 09 03:13 Phonebin.tar.gz
-rw-r--r-- 1 804 804 436921 Nov 09 03:13 Picasa.tar.gz
-rw-r--r-- 1 804 804 255436 Nov 09 03:13 The Big Picture.tar.gz
-rw-r--r-- 1 804 804 479633 Nov 09 03:12 flickr.tar.gz
-rw-r--r-- 1 804 804 2351538 Nov 09 03:12 iPhoto.tar.gz
-rw-r--r-- 1 804 804 21191 Nov 09 03:13 widelec.org.tar.gz
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/Phonebin.tar.gz
DELETE FROM temp1 WHERE id = 4
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/Picasa.tar.gz
DELETE FROM temp1 WHERE id = 5
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/flickr.tar.gz
DELETE FROM temp1 WHERE id = 7
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/iPhoto.tar.gz
DELETE FROM temp1 WHERE id = 8
SELECT mirr_add_bypath(?, ?); <-- 1, addons/plugins/pictures/widelec.org.tar.gz
DELETE FROM temp1 WHERE id = 9
mirror.netcologne.de: committing ftp dir addons/plugins/pictures
The file is seen via FTP (first half of the log), but no action taken when it comes to storing the file into the
database (second part of log). Staring at the scanner, line 638 now.
msg93 (view) Author: poeml Date: 2009-12-05.12:39:36
fixed in trunk:
http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/tools/scanner.pl?
r1=7905&r2=7904&pathrev=7905
msg94 (view) Author: poeml Date: 2009-12-05.16:10:36
I'm verifying the results that the new scanner brings by running it on all
OpenOffice.org mirrors that are scanned via FTP; it looks good.
msg95 (view) Author: poeml Date: 2009-12-05.21:21:25
fixed in 2.11.2. Thanks for the report!
msg101 (view) Author: theuni Date: 2009-12-09.07:44:41
Whoops, missed the reply.
Indeed this is fixed and working great. Thanks for the quick turnaround as usual.
(end of migrated issue)
The text was updated successfully, but these errors were encountered: