IX-F importer fails on nulled ipv4 / ipv6 properties in vlan_list entries

[vegu]

Describe the bug

Currently the importer will raise an error when encountering null values for ipv6 or ipv4 properties in the vlan_list property.

Expected behavior

It's valid according to the IX-F schema, and the importer should support it by treating it the same way as if the property (ipv6 or ipv4) was not there at all.

Are there security concerns?

no

Are there privacy concerns?

no

[ccaputo]

@peeringdb/pc please make this an ASAP. It was reported by me and is an active problem. Thanks.

[mcmanuss8]

+1 clear bug

[martinhannigan]

+1

…

On Wed, Sep 14, 2022 at 10:17 mcmanuss8 ***@***.***> wrote: +1 clear bug — Reply to this email directly, view it on GitHub <#1244 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFA2YQTZFEXDYQMIQDPYJXDV6HNA7ANCNFSM6AAAAAAQMDYF4Y> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[arnoldnipper]

@vegu, please generally accept null instead of at least "".

[ccaputo]

The traceback has evolved with this update, but the problem remains.

On production, the following is reported:

ERROR: 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "/srv/www.peeringdb.com/peeringdb_server/management/commands/pdb_ixf_ixp_member_import.py", line 278, in handle
    success = importer.update(
  File "/srv/www.peeringdb.com/peeringdb_server/ixf.py", line 501, in update
    data = self.fetch(ixlan.ixf_ixp_member_list_url, timeout=timeout)
  File "/srv/www.peeringdb.com/peeringdb_server/ixf.py", line 212, in fetch
    data = self.sanitize(data)
  File "/srv/www.peeringdb.com/peeringdb_server/ixf.py", line 451, in sanitize
    ipv6 = vlans[0].get("ipv6", {}).get("address")
AttributeError: 'NoneType' object has no attribute 'get'

While on beta, with the new code, this is reported:

ERROR: 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/srv/www.peeringdb.com/peeringdb_server/management/commands/pdb_ixf_ixp_member_import.py", line 278, in handle
    success = importer.update(
  File "/srv/www.peeringdb.com/peeringdb_server/ixf.py", line 510, in update
    data = self.fetch(ixlan.ixf_ixp_member_list_url, timeout=timeout)
  File "/srv/www.peeringdb.com/peeringdb_server/ixf.py", line 212, in fetch
    data = self.sanitize(data)
  File "/srv/www.peeringdb.com/peeringdb_server/ixf.py", line 446, in sanitize
    connection_list = self.match_vlans_across_connections(
  File "/srv/www.peeringdb.com/peeringdb_server/ixf.py", line 318, in match_vlans_across_connections
    cxns_that_match = self.find_connections_that_match(
  File "/srv/www.peeringdb.com/peeringdb_server/ixf.py", line 290, in find_connections_that_match
    if self.connections_match(connection, connection2):
  File "/srv/www.peeringdb.com/peeringdb_server/ixf.py", line 282, in connections_match
    if_list_1 = self.get_if_speed_list(connection1)
  File "/srv/www.peeringdb.com/peeringdb_server/ixf.py", line 270, in get_if_speed_list
    for if_entry in connection.get("if_list", []):
TypeError: 'NoneType' object is not iterable

[ccaputo]

An if_list of:

"if_list": null

rather than the normal likes of:

        "if_list": [
                {
                        "switch_id":0,
                        "if_speed":10000
                }
        ],

seems to be tripping up the importer.

This null passes the https://ixpdb.euro-ix.net/en/validator/ validator.

[ccaputo]

This new code worked successfully on beta this 0000 UTC run of the Importer. Nice work!