Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NDB: lost sync in apply() (due 'File exists'?) when trying to bring up an interface inside a network namespace #628

Closed
rytilahti opened this issue Aug 12, 2019 · 21 comments

Comments

@rytilahti
Copy link

Hi and thanks for the great library!

I have been experimenting with the NDB interface and network namespaces, and I'm currently struggling to make the network interface inside a netns to go up.

As similar procedure works fine with the "IPDB-way", I think it is a bug, but I couldn't figure out where should I start looking to fix it.

P.S. A quick side-question, is it safe to use multiple context managers, ala with ndb.interfaces[peer_dict] as peer_veth, nbd.interfaces[root_ns_if] as root_veth:?

NDB

interface on main namespace

1272: prcf30af0d-r@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a6:6f:4e:64:dd:36 brd ff:ff:ff:ff:ff:ff link-netns prcf30af0d
    inet 192.168.233.0/31 scope global prcf30af0d-r
       valid_lft forever preferred_lft forever
    inet6 fe80::a46f:4eff:fe64:dd36/64 scope link 
       valid_lft forever preferred_lft forever

Interface inside the NS

  • Note the smallish index number, potentially related as IPDB assigns a sequantial index?
  • On the main namespace that index is reserved for the ethernet adapter.
2: prcf30af0d-p@if1272: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 36:65:27:22:ae:76 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.233.1/31 scope global prcf30af0d-p
       valid_lft forever preferred_lft forever

Debug log

DEBUG:pyroute2.ndb.139946316556880.view.interfaces:cache del (('target', 'localhost'), ('tflags', 0), ('index', 1272))
DEBUG:pyroute2.ndb.139946316556880.view.interfaces:cache add (('target', 'prcf30af0d'), ('tflags', 0), ('index', 2))
INFO:__main__:setting peer iface prcf30af0d-p up with 192.168.233.1/31
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:commit: [(1565646783.800148, 'invalid'), (1565646783.8018253, 'system')]
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:init
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:complete key OrderedDict([('target', 'prcf30af0d'), ('tflags', 0), ('index', 2)]) from table interfaces_139946275317232
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:got OrderedDict([('target', 'prcf30af0d'), ('tflags', 0), ('index', 2)])
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('prcf30af0d', 0, 0, 1, 2, 4098, 0, '36:65:27:22:ae:76', 'ff:ff:ff:ff:ff:ff', 'prcf30af0d-p', 1500, 1272, 'noop', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 0, 1, 0, None, 0, 65535, 65536, 0, 1, 'down', 'veth', None)
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot interfaces_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot addresses_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot neighbours_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot routes_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot nh_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot rules_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot netns_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot p2p_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot ifinfo_bridge_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot ifinfo_bond_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot ifinfo_vlan_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot ifinfo_vxlan_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot ifinfo_gre_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot ifinfo_vrf_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot ifinfo_vti_139946275317232
DEBUG:pyroute2.ndb.139946316556880.schema:create snapshot ifinfo_vti6_139946275317232
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:apply: [(1565646783.800148, 'invalid'), (1565646783.8018253, 'system')]
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('prcf30af0d', 0, 0, 1, 2, 4098, 0, '36:65:27:22:ae:76', 'ff:ff:ff:ff:ff:ff', 'prcf30af0d-p', 1500, 1272, 'noop', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 0, 1, 0, None, 0, 65535, 65536, 0, 1, 'down', 'veth', None)
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:apply req: {'target': 'prcf30af0d', 'tflags': 0, 'index': 2, 'state': 'up', 'master': None}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:apply idx_req: {'target': 'prcf30af0d', 'tflags': 0, 'index': 2}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:apply state: system
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:run set ({'target': 'prcf30af0d', 'tflags': 0, 'index': 2, 'state': 'up', 'master': None})
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:stats: apply set {objid 139946275317232, wtime 0.4, mqsize 4, nqsize 0}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_rtnl: {'length': 1360, 'type': 16, 'flags': 0, 'sequence_number': 0, 'pid': 0, 'error': None, 'stats': Stats(qsize=0, delta=0, delay=0)}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('prcf30af0d', 0, 0, 1, 2, 69635, 1, '36:65:27:22:ae:76', 'ff:ff:ff:ff:ff:ff', 'prcf30af0d-p', 1500, 1272, 'noqueue', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 1, 2, 0, None, 0, 65535, 65536, 1, 1, 'up', 'veth', None)
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('prcf30af0d', 0, 0, 1, 2, 69635, 1, '36:65:27:22:ae:76', 'ff:ff:ff:ff:ff:ff', 'prcf30af0d-p', 1500, 1272, 'noqueue', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 1, 2, 0, None, 0, 65535, 65536, 1, 1, 'up', 'veth', None)
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check: [(1565646783.800148, 'invalid'), (1565646783.844721, 'system')]
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check: True
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:checked
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:stats: 139946275317232 pass
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:init
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:complete key {'target': 'localhost', 'address': '192.168.233.1', 'prefixlen': 31} from table addresses
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_rtnl: {'length': 1360, 'type': 16, 'flags': 0, 'sequence_number': 0, 'pid': 0, 'error': None, 'stats': Stats(qsize=0, delta=0, delay=0)}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('localhost', 0, 0, 1, 1272, 69699, 0, 'a6:6f:4e:64:dd:36', 'ff:ff:ff:ff:ff:ff', 'prcf30af0d-r', 1500, 2, 'noqueue', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 1, 2, 0, None, 0, 65535, 65536, 1, 1, 'up', 'veth', None)
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_rtnl: {'length': 1360, 'type': 16, 'flags': 0, 'sequence_number': 0, 'pid': 0, 'error': None, 'stats': Stats(qsize=0, delta=0, delay=0)}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('prcf30af0d', 0, 0, 1, 2, 69699, 0, '36:65:27:22:ae:76', 'ff:ff:ff:ff:ff:ff', 'prcf30af0d-p', 1500, 1272, 'noqueue', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 1, 2, 0, None, 0, 65535, 65536, 1, 1, 'up', 'veth', None)
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:got none
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:apply: [(1565646783.8450475, 'invalid')]
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: None
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:apply req: {'target': 'prcf30af0d', 'prefixlen': 31, 'IFA_ADDRESS': '192.168.233.1', 'address': '192.168.233.1'}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:apply idx_req: {'target': 'prcf30af0d', 'prefixlen': 31, 'IFA_ADDRESS': '192.168.233.1'}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:apply state: invalid
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:run add ({'target': 'prcf30af0d', 'address': '192.168.233.1', 'prefixlen': 31, 'index': 2})
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:stats: apply add {objid 139946275357616, wtime 0.0, mqsize 0, nqsize 0}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('prcf30af0d', 0, 2, 31, 128, 0, 2, '192.168.233.1', '192.168.233.1', 'prcf30af0d-p', None, None, None, 128)
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check: [(1565646783.8450475, 'invalid'), (1565646783.854934, 'system')]
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check changed: {'target'}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check failed
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:run add ({'target': 'prcf30af0d', 'address': '192.168.233.1', 'prefixlen': 31, 'index': 2})
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:error: (17, 'File exists')
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:ignore error 17 for {'target': 'localhost', 'address': '192.168.233.1', 'prefixlen': 31, 'tflags': 0, 'family': 2, 'flags': 128, 'scope': 0, 'index': 2, 'local': '192.168.233.1', 'label': 'prcf30af0d-p', 'broadcast': None, 'anycast': None, 'multicast': None}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:run fallback set ({'target': 'prcf30af0d', 'address': '192.168.233.1', 'prefixlen': 31, 'index': 2})
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:stats: apply add {objid 139946275357616, wtime 0.1, mqsize 1, nqsize 0}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('prcf30af0d', 0, 2, 31, 128, 0, 2, '192.168.233.1', '192.168.233.1', 'prcf30af0d-p', None, None, None, 128)
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check: [(1565646783.8450475, 'invalid'), (1565646783.8588128, 'system')]
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check changed: {'target'}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check failed
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:run add ({'target': 'prcf30af0d', 'address': '192.168.233.1', 'prefixlen': 31, 'index': 2})
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:error: (17, 'File exists')
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:ignore error 17 for {'target': 'localhost', 'address': '192.168.233.1', 'prefixlen': 31, 'tflags': 0, 'family': 2, 'flags': 128, 'scope': 0, 'index': 2, 'local': '192.168.233.1', 'label': 'prcf30af0d-p', 'broadcast': None, 'anycast': None, 'multicast': None}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:run fallback set ({'target': 'prcf30af0d', 'address': '192.168.233.1', 'prefixlen': 31, 'index': 2})
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:stats: apply add {objid 139946275357616, wtime 0.2, mqsize 0, nqsize 0}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('prcf30af0d', 0, 2, 31, 128, 0, 2, '192.168.233.1', '192.168.233.1', 'prcf30af0d-p', None, None, None, 128)
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check: [(1565646783.8450475, 'invalid'), (1565646783.9650478, 'system')]
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check changed: {'target'}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check failed
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:run add ({'target': 'prcf30af0d', 'address': '192.168.233.1', 'prefixlen': 31, 'index': 2})
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:error: (17, 'File exists')

<I snipped off some more tries from here>

DEBUG:pyroute2.ndb.139946316556880.rtnl_object:run set ({'target': 'prcf30af0d', 'tflags': 0, 'index': 2, 'state': 'down', 'master': None})
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:stats: apply set {objid 139946275002032, wtime 0.4, mqsize 4, nqsize 0}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_rtnl: {'length': 1360, 'type': 16, 'flags': 0, 'sequence_number': 0, 'pid': 0, 'error': None, 'stats': Stats(qsize=0, delta=0, delay=0)}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('prcf30af0d', 0, 0, 1, 2, 4098, 1, '36:65:27:22:ae:76', 'ff:ff:ff:ff:ff:ff', 'prcf30af0d-p', 1500, 1272, 'noqueue', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 0, 3, 0, None, 0, 65535, 65536, 1, 2, 'down', 'veth', None)
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('prcf30af0d', 0, 0, 1, 2, 4098, 0, '36:65:27:22:ae:76', 'ff:ff:ff:ff:ff:ff', 'prcf30af0d-p', 1500, 1272, 'noop', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 0, 1, 0, None, 0, 65535, 65536, 0, 1, 'down', 'veth', None)
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check: [(1565646783.808394, 'invalid'), (1565646798.4870124, 'system')]
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:check: True
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:checked
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:stats: 139946275002032 pass
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_rtnl: {'length': 1360, 'type': 16, 'flags': 0, 'sequence_number': 0, 'pid': 0, 'error': None, 'stats': Stats(qsize=0, delta=-1, delay=0)}
DEBUG:pyroute2.ndb.139946316556880.rtnl_object:load_sql: ('localhost', 0, 0, 1, 1272, 4099, 0, 'a6:6f:4e:64:dd:36', 'ff:ff:ff:ff:ff:ff', 'prcf30af0d-r', 1500, 2, 'noqueue', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 0, 3, 0, None, 0, 65535, 65536, 1, 2, 'up', 'veth', None)
Traceback (most recent call last):
  File "test_ndb.py", line 65, in <module>
    create_namespace_nbd()
  File "test_ndb.py", line 39, in create_namespace_nbd
    _LOGGER.info(f"setting peer iface {peer_ns_if} up with {peer_ip_w_cidr}")
  File "/code/path/lib/pyroute2/pyroute2/ndb/objects/__init__.py", line 261, in __exit__
    self.commit()
  File "/code/path/lib/pyroute2/pyroute2/ndb/objects/__init__.py", line 496, in commit
    self.apply()
  File "/code/path/lib/pyroute2/pyroute2/ndb/objects/interface.py", line 315, in apply
    super(Interface, self).apply(rollback)
  File "/code/path/lib/pyroute2/pyroute2/ndb/objects/__init__.py", line 715, in apply
    raise ret
  File "/code/path/lib/pyroute2/pyroute2/ndb/objects/interface.py", line 211, in do_add_ip
    self.ipaddr.create(spec).apply()
  File "/code/path/lib/pyroute2/pyroute2/ndb/objects/__init__.py", line 672, in apply
    raise Exception('lost sync in apply()')
Exception: lost sync in apply()

IPDB variant

  • Running the IPDB version works as expected.
  • The interface numbers are sequential here.

Main namespace

1274: pr5ddf7a81-r@if1273: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 62:f4:1a:e1:10:23 brd ff:ff:ff:ff:ff:ff link-netns pr5ddf7a81
    inet 192.168.233.0/31 scope global pr5ddf7a81-r
       valid_lft forever preferred_lft forever
    inet6 fe80::60f4:1aff:fee1:1023/64 scope link 
       valid_lft forever preferred_lft forever

Inside the namespace

1273: pr5ddf7a81-p@if1274: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d6:80:b2:5a:a0:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.233.1/31 scope global pr5ddf7a81-p
       valid_lft forever preferred_lft forever
    inet6 fe80::d480:b2ff:fe5a:a004/64 scope link 
       valid_lft forever preferred_lft forever

Test script

  • Note, this does not clean the interfaces nor the namespace afterwards, for testing I simple used ip -all netns delete.
from pyroute2 import IPRoute, netns, NetNS, NDB, IPDB
from pyroute2.common import uifname
import logging

logging.basicConfig(level=logging.DEBUG)

_LOGGER = logging.getLogger(__name__)

root_ip, peer_ip = "192.168.233.0", "192.168.233.1"
root_ip_w_cidr = f"{root_ip}/31"
peer_ip_w_cidr = f"{peer_ip}/31"

nsname = uifname()

root_ns_if = f"{nsname}-r"
peer_ns_if = f"{nsname}-p"


def create_namespace_nbd():
    ndb = NDB(debug=True)

    ndb.sources.add(netns=nsname)

    ndb.interfaces.create(
        ifname=root_ns_if, kind="veth", peer={"ifname": peer_ns_if, "net_ns_fd": nsname}
    ).commit()

    with ndb.interfaces[root_ns_if] as root_veth:
        root_veth.add_ip(root_ip_w_cidr)
        root_veth["state"] = "up"
        _LOGGER.info(f"setting root iface {root_ns_if} up with {root_ip_w_cidr}")

    # need to use net_ns_fd to locate the peer interface
    peer_dict = {"net_ns_fd": nsname, "ifname": peer_ns_if}
    with ndb.interfaces[peer_dict] as peer_veth:
        peer_veth.add_ip(peer_ip_w_cidr)
        peer_veth["state"] = "up"
        _LOGGER.info(f"setting peer iface {peer_ns_if} up with {peer_ip_w_cidr}")


def create_namespace_ipr():
    ipdb = IPDB()

    ns = NetNS(nsname)

    ipdb.create(ifname=root_ns_if, kind="veth", peer=peer_ns_if).commit()
    with ipdb.interfaces[peer_ns_if] as i:
        i.net_ns_fd = nsname

    with ipdb.interfaces[root_ns_if] as root_veth:
        root_veth.add_ip(root_ip_w_cidr)
        root_veth.up()

    ipdb_ns = IPDB(nl=ns)

    with ipdb_ns.interfaces[peer_ns_if] as peer_veth:
        peer_veth.add_ip(peer_ip_w_cidr)
        peer_veth.up()

    _LOGGER.info(f"created successfully.")
    ipdb_ns.release()
    ns.close()


# create_namespace_nbd()
create_namespace_ipr()
@svinota
Copy link
Owner

svinota commented Aug 20, 2019

Sorry for the delay.

Please try

# notice "target" instead of "net_ns_fd" here:
peer_dict = {"target": nsname, "ifname": peer_ns_if}

Regarding the question:

is it safe to use multiple context managers

must be OK, if not — pls open a ticket

@afics
Copy link

afics commented Nov 16, 2019

@svinota for me this still happens with your suggested change (net_ns_fd -> target).

@mawiegand
Copy link
Contributor

mawiegand commented Jan 14, 2020

Any news here?

I run into the same error if I try to execute the following:

#!/usr/bin/python3
 
from pyroute2 import NDB
 
ndb = NDB(debug=True,log="on")
 
ndb.sources.add(netns="ns1")
 
# move interface to namespace "ns1"
dev = ndb.interfaces[{"ifname": "enp0s3"}]
dev = dev.set("target", "ns1").commit()
 
# add ip address
dev = dev.add_ip("192.168.42.21/24")
dev = dev.commit()

@svinota
Copy link
Owner

svinota commented Jan 14, 2020

(/me back from long absence)

Found the issue, will be fixed on Thursday.

svinota added a commit that referenced this issue Jan 16, 2020
A proper fix is on the way.

Bug-Url: #628
@svinota
Copy link
Owner

svinota commented Jan 16, 2020

Workaround is merged, pls check if it helps. But the proper fix is on the way.

@mawiegand
Copy link
Contributor

Workaround is merged, pls check if it helps. But the proper fix is on the way.

Seems still to be broken:

Traceback (most recent call last):
  File "./pyroute2_test.py", line 10, in <module>
    dev = ndb.interfaces[{"ifname": "enp0s3"}]
  File "/home/mawiegand/pyroute2/pyroute2/ndb/main.py", line 299, in __getitem__
    match_pairs=match_pairs)
  File "/home/mawiegand/pyroute2/pyroute2/ndb/objects/interface.py", line 150, in __init__
    super(Interface, self).__init__(*argv, **kwarg)
  File "/home/mawiegand/pyroute2/pyroute2/ndb/objects/__init__.py", line 231, in __init__
    raise KeyError('object does not exists')
KeyError: 'object does not exists'

Log:

2020-01-16 06:24:07,482    DEBUG pyroute2.ndb.140602995765824.sources.ns1: init                                                                                                                                                                                       [22/1939]
2020-01-16 06:24:07,482    DEBUG pyroute2.ndb.140602995765824.sources.ns1: starting the source
2020-01-16 06:24:07,483    DEBUG pyroute2.ndb.140602995765824.sources.ns1: connecting                                                                                                                                                                                         
2020-01-16 06:24:07,489    DEBUG pyroute2.ndb.140602995765824.sources.ns1: loading                                                                                                                                                                                            
2020-01-16 06:24:07,500    DEBUG pyroute2.ndb.140602995765824.sources.ns1: running                                                                                                                                                                                            
2020-01-16 06:24:07,500    DEBUG pyroute2.ndb.140602995765824.rtnl_object: init                                                                                                                                                                                               
2020-01-16 06:24:07,500    DEBUG pyroute2.ndb.140602995765824.rtnl_object: complete key {'ifname': 'enp0s3'} from table interfaces                                                                                                                                            
2020-01-16 06:24:07,500    DEBUG pyroute2.ndb.140602995765824.rtnl_object: got {'ifname': 'enp0s3', 'target': 'localhost', 'tflags': 0, 'index': 2}                                                                                                                           
2020-01-16 06:24:07,501    DEBUG pyroute2.ndb.140602995765824.rtnl_object: load_sql: ('localhost', 0, 0, 1, 2, 4098, 0, '08:00:27:61:87:28', 'ff:ff:ff:ff:ff:ff', 'enp0s3', 1500, None, 'noop', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 0, 3, None, None, 0, 65535,
65536, 1, 2, 'down', None, None)
2020-01-16 06:24:07,501    DEBUG pyroute2.ndb.140602995765824.view.interfaces: cache add (('target', 'localhost'), ('tflags', 0), ('index', 2))                                                                                                                               
2020-01-16 06:24:07,501    DEBUG pyroute2.ndb.140602995765824.rtnl_object: commit: [(1579177447.5005624, 'invalid'), (1579177447.5011153, 'system'), (1579177447.5012302, 'setns')]                                                                                           
2020-01-16 06:24:07,501    DEBUG pyroute2.ndb.140602995765824.rtnl_object: init
2020-01-16 06:24:07,501    DEBUG pyroute2.ndb.140602995765824.rtnl_object: complete key OrderedDict([('target', 'localhost'), ('tflags', 0), ('index', 2)]) from table interfaces_140602968230464                                                                             
2020-01-16 06:24:07,501    DEBUG pyroute2.ndb.140602995765824.rtnl_object: got OrderedDict([('target', 'localhost'), ('tflags', 0), ('index', 2)])                                                                                                                            
2020-01-16 06:24:07,501    DEBUG pyroute2.ndb.140602995765824.rtnl_object: load_sql: ('localhost', 0, 0, 1, 2, 4098, 0, '08:00:27:61:87:28', 'ff:ff:ff:ff:ff:ff', 'enp0s3', 1500, None, 'noop', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 0, 3, None, None, 0, 65535,
65536, 1, 2, 'down', None, None)
2020-01-16 06:24:07,502    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot interfaces_140602968230464
2020-01-16 06:24:07,503    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot addresses_140602968230464
2020-01-16 06:24:07,503    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot neighbours_140602968230464
2020-01-16 06:24:07,503    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot routes_140602968230464
2020-01-16 06:24:07,503    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot nh_140602968230464
2020-01-16 06:24:07,504    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot rules_140602968230464
2020-01-16 06:24:07,504    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot netns_140602968230464
2020-01-16 06:24:07,504    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot p2p_140602968230464
2020-01-16 06:24:07,504    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot ifinfo_bridge_140602968230464
2020-01-16 06:24:07,505    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot ifinfo_bond_140602968230464
2020-01-16 06:24:07,505    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot ifinfo_vlan_140602968230464
2020-01-16 06:24:07,506    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot ifinfo_vxlan_140602968230464
2020-01-16 06:24:07,506    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot ifinfo_gre_140602968230464
2020-01-16 06:24:07,506    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot ifinfo_vrf_140602968230464
2020-01-16 06:24:07,507    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot ifinfo_vti_140602968230464
2020-01-16 06:24:07,507    DEBUG pyroute2.ndb.140602995765824.schema: create snapshot ifinfo_vti6_140602968230464
2020-01-16 06:24:07,509    DEBUG pyroute2.ndb.140602995765824.rtnl_object: apply: [(1579177447.5005624, 'invalid'), (1579177447.5011153, 'system'), (1579177447.5012302, 'setns')]
2020-01-16 06:24:07,510    DEBUG pyroute2.ndb.140602995765824.rtnl_object: load_sql: ('localhost', 0, 0, 1, 2, 4098, 0, '08:00:27:61:87:28', 'ff:ff:ff:ff:ff:ff', 'enp0s3', 1500, None, 'noop', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 0, 3, None, None, 0, 65535,
65536, 1, 2, 'down', None, None)
2020-01-16 06:24:07,510    DEBUG pyroute2.ndb.140602995765824.rtnl_object: apply req: {'target': 'localhost', 'tflags': 0, 'index': 2, 'net_ns_fd': 'ns1'}
2020-01-16 06:24:07,510    DEBUG pyroute2.ndb.140602995765824.rtnl_object: apply idx_req: {'target': 'localhost', 'tflags': 0, 'index': 2}
2020-01-16 06:24:07,510    DEBUG pyroute2.ndb.140602995765824.rtnl_object: apply state: setns
2020-01-16 06:24:07,510    DEBUG pyroute2.ndb.140602995765824.rtnl_object: run set ({'target': 'localhost', 'tflags': 0, 'index': 2, 'net_ns_fd': 'ns1'})
2020-01-16 06:24:07,521    DEBUG pyroute2.ndb.140602995765824.rtnl_object: stats: apply set {objid 140602968230464, wtime 0.0, mqsize 0, nqsize 0}
2020-01-16 06:24:07,526    DEBUG pyroute2.ndb.140602995765824.rtnl_object: load_sql: ('localhost', 0, 0, 1, 2, 4098, 0, '08:00:27:61:87:28', 'ff:ff:ff:ff:ff:ff', 'enp0s3', 1500, None, 'noop', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 0, 3, None, None, 0, 65535,
65536, 1, 2, 'down', None, None)
2020-01-16 06:24:07,526    DEBUG pyroute2.ndb.140602995765824.rtnl_object: check: [(1579177447.5005624, 'invalid'), (1579177447.5011153, 'system'), (1579177447.5012302, 'setns')]
2020-01-16 06:24:07,526    DEBUG pyroute2.ndb.140602995765824.rtnl_object: check state: False
2020-01-16 06:24:07,526    DEBUG pyroute2.ndb.140602995765824.rtnl_object: check failed
2020-01-16 06:24:07,526    DEBUG pyroute2.ndb.140602995765824.rtnl_object: run set ({'target': 'localhost', 'tflags': 0, 'index': 2, 'net_ns_fd': 'ns1'})
2020-01-16 06:24:07,526    DEBUG pyroute2.ndb.140602995765824.rtnl_object: error: (19, 'No such device')
2020-01-16 06:24:07,526    DEBUG pyroute2.ndb.140602995765824.rtnl_object: ignore error 19 for {'flags': 4098, 'state': 'down', 'ifname': 'enp0s3', 'target': 'localhost', 'tflags': 0, 'index': 2, 'family': 0, 'ifi_type': 1, 'change': 0, 'address': '08:00:27:61:87:28', 'b
roadcast': 'ff:ff:ff:ff:ff:ff', 'mtu': 1500, 'link': None, 'qdisc': 'noop', 'master': None, 'txqlen': 1000, 'linkmode': 0, 'net_ns_pid': None, 'ifalias': None, 'num_vf': None, 'group': 0, 'ext_mask': None, 'promiscuity': 0, 'num_tx_queues': 1, 'num_rx_queues': 1, 'carrie
r': 0, 'carrier_changes': 3, 'link_netnsid': None, 'phys_port_name': None, 'proto_down': 0, 'gso_max_segs': 65535, 'gso_max_size': 65536, 'carrier_up_count': 1, 'carrier_down_count': 2, 'kind': None, 'slave_kind': None, 'net_ns_fd': 'ns1'}
2020-01-16 06:24:07,527    DEBUG pyroute2.ndb.140602995765824.rtnl_object: stats: apply set {objid 140602968230464, wtime 0.2, mqsize 2, nqsize 0}
2020-01-16 06:24:07,528    DEBUG pyroute2.ndb.140602995765824.rtnl_object: load_rtnl: {'length': 580, 'type': 17, 'flags': 0, 'sequence_number': 0, 'pid': 0, 'error': None, 'stats': Stats(qsize=0, delta=0, delay=0)}
2020-01-16 06:24:07,530    DEBUG pyroute2.ndb.140602995765824.rtnl_object: load_sql: None
2020-01-16 06:24:07,530    DEBUG pyroute2.ndb.140602995765824.rtnl_object: check: [(1579177447.5005624, 'invalid'), (1579177447.5011153, 'system'), (1579177447.5012302, 'setns'), (1579177447.5283737, 'invalid')]
2020-01-16 06:24:07,530    DEBUG pyroute2.ndb.140602995765824.rtnl_object: check: True
2020-01-16 06:24:07,530    DEBUG pyroute2.ndb.140602995765824.rtnl_object: checked
2020-01-16 06:24:07,530    DEBUG pyroute2.ndb.140602995765824.rtnl_object: stats: 140602968230464 pass
2020-01-16 06:24:07,530    DEBUG pyroute2.ndb.140602995765824.rtnl_object: commit: [(1579177447.5005624, 'invalid'), (1579177447.5011153, 'system'), (1579177447.5012302, 'setns'), (1579177447.5283737, 'invalid')]
2020-01-16 06:24:07,530    DEBUG pyroute2.ndb.140602995765824.rtnl_object: apply: [(1579177447.5005624, 'invalid'), (1579177447.5011153, 'system'), (1579177447.5012302, 'setns'), (1579177447.5283737, 'invalid')]
2020-01-16 06:24:07,530    DEBUG pyroute2.ndb.140602995765824.rtnl_object: load_sql: None
2020-01-16 06:24:07,531    DEBUG pyroute2.ndb.140602995765824.rtnl_object: apply req: {'target': 'localhost', 'tflags': 0, 'index': 2}
2020-01-16 06:24:07,531    DEBUG pyroute2.ndb.140602995765824.rtnl_object: apply idx_req: {'target': 'localhost', 'tflags': 0, 'index': 2}
2020-01-16 06:24:07,531    DEBUG pyroute2.ndb.140602995765824.rtnl_object: apply state: invalid
2020-01-16 06:24:07,531    DEBUG pyroute2.ndb.140602995765824.rtnl_object: run add ({'flags': 4098, 'state': 'down', 'ifname': 'enp0s3', 'target': 'localhost', 'tflags': 0, 'index': 2, 'family': 0, 'ifi_type': 1, 'change': 0, 'address': '08:00:27:61:87:28', 'broadcast':
'ff:ff:ff:ff:ff:ff', 'mtu': 1500, 'qdisc': 'noop', 'txqlen': 1000, 'linkmode': 0, 'group': 0, 'promiscuity': 0, 'num_tx_queues': 1, 'num_rx_queues': 1, 'carrier': 0, 'carrier_changes': 3, 'proto_down': 0, 'gso_max_segs': 65535, 'gso_max_size': 65536, 'carrier_up_count':
1, 'carrier_down_count': 2, 'net_ns_fd': 'ns1'})
2020-01-16 06:24:07,532    DEBUG pyroute2.ndb.140602995765824.rtnl_object: error: (95, 'Operation not supported')
2020-01-16 06:24:07,532    DEBUG pyroute2.ndb.140602995765824.rtnl_object: init
2020-01-16 06:24:07,532    DEBUG pyroute2.ndb.140602995765824.rtnl_object: complete key {'flags': 4098, 'state': 'down', 'ifname': 'enp0s3', 'target': 'localhost', 'tflags': 0, 'index': 2, 'family': 0, 'ifi_type': 1, 'change': 0, 'address': '08:00:27:61:87:28', 'broadcas
t': 'ff:ff:ff:ff:ff:ff', 'mtu': 1500, 'link': None, 'qdisc': 'noop', 'txqlen': 1000, 'linkmode': 0, 'net_ns_pid': None, 'ifalias': None, 'num_vf': None, 'group': 0, 'ext_mask': None, 'promiscuity': 0, 'num_tx_queues': 1, 'num_rx_queues': 1, 'carrier': 0, 'carrier_changes
': 3, 'link_netnsid': None, 'phys_port_name': None, 'proto_down': 0, 'gso_max_segs': 65535, 'gso_max_size': 65536, 'carrier_up_count': 1, 'carrier_down_count': 2, 'kind': None, 'slave_kind': None, 'net_ns_fd': 'ns1'} from table interfaces
2020-01-16 06:24:07,532    DEBUG pyroute2.ndb.140602995765824.rtnl_object: got {'flags': 4098, 'state': 'down', 'ifname': 'enp0s3', 'target': 'localhost', 'tflags': 0, 'index': 2, 'family': 0, 'ifi_type': 1, 'change': 0, 'address': '08:00:27:61:87:28', 'broadcast': 'ff:f
f:ff:ff:ff:ff', 'mtu': 1500, 'link': None, 'qdisc': 'noop', 'txqlen': 1000, 'linkmode': 0, 'net_ns_pid': None, 'ifalias': None, 'num_vf': None, 'group': 0, 'ext_mask': None, 'promiscuity': 0, 'num_tx_queues': 1, 'num_rx_queues': 1, 'carrier': 0, 'carrier_changes': 3, 'li
nk_netnsid': None, 'phys_port_name': None, 'proto_down': 0, 'gso_max_segs': 65535, 'gso_max_size': 65536, 'carrier_up_count': 1, 'carrier_down_count': 2, 'kind': None, 'slave_kind': None, 'net_ns_fd': 'ns1'}
2020-01-16 06:24:17,377    DEBUG pyroute2.ndb.140602995765824.sources.ns1: sync
2020-01-16 06:24:17,377    DEBUG pyroute2.ndb.140602995765824.sources.ns1: flush DB for the target
2020-01-16 06:24:17,380    DEBUG pyroute2.ndb.140602995765824.sources.ns1: stopped
2020-01-16 06:24:17,382     INFO pyroute2.ndb.140602995765824.sources.localhost: source shutdown
2020-01-16 06:24:17,383    DEBUG pyroute2.ndb.140602995765824.sources.localhost: sync
2020-01-16 06:24:17,383    DEBUG pyroute2.ndb.140602995765824.sources.localhost: shutdown handled by the main thread
2020-01-16 06:24:17,383    DEBUG pyroute2.ndb.140602995765824.sources.localhost: stopped
2020-01-16 06:24:17,384    DEBUG pyroute2.ndb.140602995765824.main: flush DB for the target localhost
2020-01-16 06:24:17,386     INFO pyroute2.ndb.140602995765824.sources.localhost/netns: source shutdown
2020-01-16 06:24:17,386    DEBUG pyroute2.ndb.140602995765824.sources.localhost/netns: sync
2020-01-16 06:24:17,387    DEBUG pyroute2.ndb.140602995765824.sources.localhost/netns: shutdown handled by the main thread
2020-01-16 06:24:17,387    DEBUG pyroute2.ndb.140602995765824.sources.localhost/netns: stopped
2020-01-16 06:24:17,387    DEBUG pyroute2.ndb.140602995765824.main: flush DB for the target localhost/netns
2020-01-16 06:24:17,388     INFO pyroute2.ndb.140602995765824.sources.ns1: source shutdown
2020-01-16 06:24:17,388    DEBUG pyroute2.ndb.140602995765824.main: flush DB for the target ns1

@svinota
Copy link
Owner

svinota commented Jan 16, 2020

Thanks, analyzing the log

@svinota
Copy link
Owner

svinota commented Jan 16, 2020

@mawiegand may I look at the code you run? I believe you tried to move a real interace between netns, but for me it works. Probably it's not a bug but a poor documentation that should be updated.

@mawiegand
Copy link
Contributor

@mawiegand may I look at the code you run? I believe you tried to move a real interace between netns, but for me it works. Probably it's not a bug but a poor documentation that should be updated.

I used this code snippet:

Any news here?

I run into the same error if I try to execute the following:

#!/usr/bin/python3
 
from pyroute2 import NDB
 
ndb = NDB(debug=True,log="on")
 
ndb.sources.add(netns="ns1")
 
# move interface to namespace "ns1"
dev = ndb.interfaces[{"ifname": "enp0s3"}]
dev = dev.set("target", "ns1").commit()
 
# add ip address
dev = dev.add_ip("192.168.42.21/24")
dev = dev.commit()

And yes it is a real interface. Do I need to move real interfaces differently?

@svinota
Copy link
Owner

svinota commented Jan 16, 2020

Pls try that.

# dev = dev.set("target", "ns1").commit()
dev = dev.set("net_ns_fd", "ns1").commit()

@svinota
Copy link
Owner

svinota commented Jan 16, 2020

I have to finally do something with that explicit target assignment and fix the syntax to be intuitive.

@svinota
Copy link
Owner

svinota commented Jan 17, 2020

No, actually it already works on the master branch.

Could you please check that you run the code on the master and tell me which kernel and Python versions do you use.

@mawiegand
Copy link
Contributor

Sorry, for the late response. I was very busy last weeks.

I am using Kernel 5.5.1 and Python 3.8.1.

I changed my test code from

#!/usr/bin/python3
 
from pyroute2 import NDB
 
ndb = NDB(debug=True,log="on")
 
ndb.sources.add(netns="ns1")
 
# move interface to namespace "ns1"
dev = ndb.interfaces[{"ifname": "enp0s3"}]
dev = dev.set("target", "ns1").commit()
 
# add ip address
dev = dev.add_ip("192.168.42.21/24")
dev = dev.commit()

to

#!/usr/bin/python3
 
from pyroute2 import NDB
 
ndb = NDB(debug=True,log="on")
 
ndb.sources.add(netns="ns1")
 
# move interface to namespace "ns1"
dev = ndb.interfaces[{"ifname": "enp0s3"}]
dev = dev.set("target", "ns1").commit()
 
# get updated interface
dev = ndb.interfaces[{"ifname": "enp0s3"}]

# add ip address
dev = dev.add_ip("192.168.42.21/24")
dev = dev.commit()

Now it's working.

But calling dev.del_ip("192.168.42.21/24").commit() on an interface inside a namespace does still not work.

@svinota
Copy link
Owner

svinota commented Feb 4, 2020

Got it. Thanks a lot for the case, fixing.

@mawiegand
Copy link
Contributor

Creating a route inside a namespace with

ndb.routes.create(target=nsname, dst=target_network, gateway=next_hop).commit()

leads to same behavior. But the route is created on the system (verified via ip r).

svinota added a commit that referenced this issue Feb 5, 2020
svinota added a commit that referenced this issue Feb 5, 2020
svinota added a commit that referenced this issue Feb 5, 2020
Accept 'x/x' syntax

Bug-Url: #628
@svinota
Copy link
Owner

svinota commented Feb 5, 2020

A bunch of fixes has arrived.

@mawiegand
Copy link
Contributor

mawiegand commented Feb 20, 2020

A bunch of fixes has arrived.

Thanks!

Creating a route inside a namespace with

ndb.routes.create(target=nsname, dst=target_network, gateway=next_hop).commit()

leads to same behavior. But the route is created on the system (verified via ip r).

Seems to be working for 'normal' routes now.
But the creation of a 'default route' with

ndb.routes.create(target=nsname, dst='default', gateway=next_hop).commit()

still leads to same behavior.

@svinota
Copy link
Owner

svinota commented Feb 20, 2020

It's weird but I have to confirm. There is something really broken. Fixing

Thanks for your patience and thanks for reporting.

@mawiegand
Copy link
Contributor

It's weird but I have to confirm. There is something really broken. Fixing

Thanks for your patience and thanks for reporting.

Thanks for your hard work. If I can assist you somehow please let me know!

But calling dev.del_ip("192.168.42.21/24").commit() on an interface inside a namespace does still not work.

del_ip() inside a namepsace is also still not working and creates a trace back with
KeyError: 'object does not exists'

In the default namespace it works without problems.

svinota added a commit that referenced this issue Feb 21, 2020
On the assignment step::

    # 'default', '0/0', '0.0.0.0/0' →
    self['dst'] = ''
    self['dst_len'] = 0

Bug-Url: #628
svinota added a commit that referenced this issue Feb 21, 2020
@svinota
Copy link
Owner

svinota commented Feb 24, 2020

Just to notify that the next chunk of fixes has arrived. The functional tests are not ready yet, working on it.

svinota added a commit that referenced this issue Feb 28, 2020
Make Basic class work both in the main and a custom netns
transparently, thus applying same tests for the 'localhost'
target as well as for netns targets.

Bug-Url: #628
svinota added a commit that referenced this issue Feb 29, 2020
@svinota
Copy link
Owner

svinota commented Mar 2, 2020

Closing the ticket. Feel free to reopen it or to create a new one in the case you hit further issues.

@svinota svinota closed this as completed Mar 2, 2020
bmwiedemann added a commit to bmwiedemann/openSUSE that referenced this issue Mar 16, 2020
https://build.opensuse.org/request/show/785076
by user dirkmueller + dimstar_suse
- update to 0.5.10:
  * general: don't use pkg_resources <svinota/pyroute2#677>
  * iproute: fix Windows support
  * netlink: provide the target field
  * ndb: use the target field from the netlink header
  * ndb: multiple SQL fixes, transactions fixed with the PostgreSQL backend
  * ndb: multiple object cache fixes <svinota/pyroute2#683>
  * ndb.schema: drop DB triggers
  * ndb.objects: fix object management within a netns <svinota/pyroute2#628>
  * ndb.objects.route: support route metrics
  * ndb.objects.route: fix default route syntax
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants