asn_get_routeviews broken

[poeml]

                                                                                                          [          ]

Issue migrated (2015-06-05) from old issue tracker http://mirrorbrain.org/issues/issue137

Title    asn_get_routeviews broken
 Priority   urgent  Status   resolved
Superseder         Nosy List floeff, poeml
Assigned To poeml  Keywords

msg476 (view) Author: floeff Date: 2014-01-27.09:40:26

It seems asn_get_routeviews | asn_import -b main is broken:

Downloading http://mirrorbrain.org/routeviews/oix-full-snapshot-latest.dat.bz2
Traceback (most recent call last):
File "/usr/bin/asn_get_routeviews", line 197, in
main()
File "/usr/bin/asn_get_routeviews", line 188, in main
for pfx, asnb, asn in pfxasn_uniq:
File "/usr/bin/asn_get_routeviews", line 132, in gen_firstuniq
for tupl in tupls:
File "/usr/bin/asn_get_routeviews", line 100, in gen_asn
for line in lines:
File "/usr/bin/asn_get_routeviews", line 82, in gen_grep
for line in lines:
File "/usr/bin/asn_get_routeviews", line 66, in gen_lines
for line in lines:
File "/usr/bin/asn_get_routeviews", line 48, in gen_cat
for item in s:
EOFError: compressed file ended before the logical end-of-stream was detected

This also leads to the following error message for mb update -a:

psycopg2.IntegrityError: null value in column "prefix" violates not-null constraint

Probably it's related to those messages:

mirrorservice.org: updating network prefix (212.219.0.0/16 -> None)
ftp.rezopole.net: updating network prefix (77.95.64.0/22 -> None)
nexcess.net: updating network prefix (208.69.120.0/21 -> None)
free.nchc.org.tw: updating network prefix (211.73.64.0/19 -> None)
ftp.novell.hu: updating network prefix (213.163.0.0/18 -> None)
etc.

msg478 (view) Author: poeml Date: 2014-01-27.20:34:30

The file is indeed broken.

bzcat oix-full-snapshot-latest.dat.bz2 > /dev/null

bzcat: Compressed file ends unexpectedly;
perhaps it is corrupted? Possible reason follows.
[...]

msg479 (view) Author: poeml Date: 2014-01-27.20:55:15

But the routing table snapshots are provided every few hours, and apparently the issue has been fixed. The
version of the file currently mirrored on mirrorbrain.org works.

Let me know if you still get database errors after re-running the command!
Peter

msg481 (view) Author: floeff Date: 2014-01-27.21:03:17

Partially indeed - the download works again. However, mb main update yields
errors still. Is this a bug in the file, or something different?

mb -b main update --all --all-mirrors

ftp.rezopole.net: updating network prefix (77.95.64.0/22 -> None)
Traceback (most recent call last):
File "/usr/bin/mb", line 1655, in
r = mirrordoctor.main()
File "/usr/lib/pymodules/python2.7/cmdln.py", line 257, in main
return self.cmd(args)
File "/usr/lib/pymodules/python2.7/cmdln.py", line 280, in cmd
retval = self.onecmd(argv)
File "/usr/lib/pymodules/python2.7/cmdln.py", line 412, in onecmd
return self._dispatch_cmd(handler, argv)
File "/usr/lib/pymodules/python2.7/cmdln.py", line 1100, in _dispatch_cmd
return handler(argv[0], opts, *args)
File "/usr/bin/mb", line 439, in do_update
mirror.prefix = res.prefix
File "", line 1, in
File "/usr/lib/pymodules/python2.7/sqlobject/main.py", line 1049, in _SO_setValue
dbValue)])
File "/usr/lib/pymodules/python2.7/sqlobject/dbconnection.py", line 523, in
_SO_update
self.sqlrepr(so.id)))
File "/usr/lib/pymodules/python2.7/sqlobject/dbconnection.py", line 349, in query
return self._runWithConnection(self._query, s)
File "/usr/lib/pymodules/python2.7/sqlobject/dbconnection.py", line 262, in
_runWithConnection
val = meth(conn, *args)
File "/usr/lib/pymodules/python2.7/sqlobject/dbconnection.py", line 346, in _query
self._executeRetry(conn, conn.cursor(), s)
File "/usr/lib/pymodules/python2.7/sqlobject/dbconnection.py", line 341, in
_executeRetry
return cursor.execute(query)
psycopg2.IntegrityError: null value in column "prefix" violates not-null constraint

msg484 (view) Author: poeml Date: 2014-01-27.21:24:16

Apparently, there is no network prefix in the current routing data snapshot which
contains 77.95.64.41. Too bad, shouldn't happen I guess. It's quite strange and would
mean that the routing data snapshot is incomplete. But, it will probably fix itself
automatically...

I can reproduce the error here for another mirror: ftp.piotrkosoft.net

As for the error spit out by "mb update", I have a local fix here that I'll commit to
SVN now, which makes mb cope like this:

mb update --all ftp.piotrkosoft.net

ftp.piotrkosoft.net: STRANGE! There's no prefix containing this hosts IP address
(178.217.221.233)...
ftp.piotrkosoft.net: STRANGE! There's no ASN containing this hosts IP address
(178.217.221.233)...
ftp.piotrkosoft.net: updating geographical coordinates (50.033 19.233 -> 52.0 20.0)

So the command completes to run.

msg485 (view) Author: poeml Date: 2014-01-27.21:31:39

the fix is here:

http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/mb/mb.py?r1=8342&r2=8343

History
         Date          User  Action             Args
2014-01-27 21:31:39 poeml  set    status: chatting -> resolved
                                    messages: + msg485
2014-01-27 21:24:16 poeml  set    messages: + msg484
2014-01-27 21:03:17 floeff set    status: resolved -> chatting
                                    messages: + msg481
2014-01-27 20:55:16 poeml  set    status: chatting -> resolved
                                    messages: + msg479
                                    status: unread -> chatting
2014-01-27 20:34:30 poeml  set    assignedto: poeml
                                    messages: + msg478
                                    nosy: + poeml
2014-01-27 09:40:26 floeff create

(end of migrated issue)