UPF Orchestration : IPAM Injection Failure in the draft stage

[rravindran123]

When i recreate a new resource type “fivegcoretopology-sample”, this will invoke all the controllers. At this stage, most likely, the IPAM injection preconditions flag are raised on PORCH, so then the patch is to delete and reapply “sample-n3-net-prefix1", this fixes the preconditions.

[henderiw]

is it possible to do a log on the ipam k logs -n Nephio-system ipam-... -f. Would like to see the logs to understand why it is not allocating the prefixes. One reason I saw was when the package was approved there was no way to update the package as it was already approved and an update could not be handled. Also this should be visible in the condition of the ipam allocation.

[rravindran123]

Sure, will share the logs with you, will have to recreate the scenario..

","labels":{"nephio.org/gateway":"true"}}}
1.6746297544239135e+09 INFO validate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "35e1cdeb-1b32-4caa-ab06-97f2b7b03815", "cr": "sample-n3-net-prefix1", "prefix": "10.1.0.1/24"}
1.674629754424011e+09 INFO cannot allocate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "35e1cdeb-1b32-4caa-ab06-97f2b7b03815", "err": "validated failed: nesting network prefixes with anything other than an aggregate prefix is not allowed, prefix nested with upf-edge1-internet of kind pool"}
1.6746297594331806e+09 INFO reconcile {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "d0b77943-e252-45df-9939-bbc57c3e5f61", "req": "default/sample-n3-net-prefix1"}
1.6746297594333088e+09 INFO allocate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "d0b77943-e252-45df-9939-bbc57c3e5f61", "alloc": {"namespacedName":{"Namespace":"default","Name":"sample-n3-net-prefix1"},"origin":"prefix","networkInstance":"sample-vpc","prefixKind":"network","addressFamily":"ipv4","prefix":"10.1.0.1/24","prefixLength":24,"network":"sample-n3-net","labels":{"nephio.org/gateway":"true"}}}
1.674629759433349e+09 INFO validate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "d0b77943-e252-45df-9939-bbc57c3e5f61", "cr": "sample-n3-net-prefix1", "prefix": "10.1.0.1/24"}
1.67462975943347e+09 INFO cannot allocate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "d0b77943-e252-45df-9939-bbc57c3e5f61", "err": "validated failed: nesting network prefixes with anything other than an aggregate prefix is not allowed, prefix nested with upf-edge1-internet of kind pool"}
1.6746297644416783e+09 INFO reconcile {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "5ef4ea6f-ee25-4e54-a425-7d95c20bb6e9", "req": "default/sample-n3-net-prefix1"}
1.674629764441803e+09 INFO allocate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "5ef4ea6f-ee25-4e54-a425-7d95c20bb6e9", "alloc": {"namespacedName":{"Namespace":"default","Name":"sample-n3-net-prefix1"},"origin":"prefix","networkInstance":"sample-vpc","prefixKind":"network","addressFamily":"ipv4","prefix":"10.1.0.1/24","prefixLength":24,"network":"sample-n3-net","labels":{"nephio.org/gateway":"true"}}}
1.6746297644418263e+09 INFO validate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "5ef4ea6f-ee25-4e54-a425-7d95c20bb6e9", "cr": "sample-n3-net-prefix1", "prefix": "10.1.0.1/24"}
1.6746297644418762e+09 INFO cannot allocate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "5ef4ea6f-ee25-4e54-a425-7d95c20bb6e9", "err": "validated failed: nesting network prefixes with anything other than an aggregate prefix is not allowed, prefix nested with upf-edge1-internet of kind pool"}
1.674629769451333e+09 INFO reconcile {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "6dde5214-1903-4769-837c-bde5074505bb", "req": "default/sample-n3-net-prefix1"}
1.674629769451493e+09 INFO allocate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "6dde5214-1903-4769-837c-bde5074505bb", "alloc": {"namespacedName":{"Namespace":"default","Name":"sample-n3-net-prefix1"},"origin":"prefix","networkInstance":"sample-vpc","prefixKind":"network","addressFamily":"ipv4","prefix":"10.1.0.1/24","prefixLength":24,"network":"sample-n3-net","labels":{"nephio.org/gateway":"true"}}}
1.6746297694515903e+09 INFO validate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "6dde5214-1903-4769-837c-bde5074505bb", "cr": "sample-n3-net-prefix1", "prefix": "10.1.0.1/24"}
1.6746297694516747e+09 INFO cannot allocate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "6dde5214-1903-4769-837c-bde5074505bb", "err": "validated failed: nesting network prefixes with anything other than an aggregate prefix is not allowed, prefix nested with upf-edge1-internet of kind pool"}
1.674629774459888e+09 INFO reconcile {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "bc70b264-1ead-4a4b-81cd-ae36cb0de5f1", "req": "default/sample-n3-net-prefix1"}
1.6746297744599931e+09 INFO allocate prefix {"controller": "ipprefix", "controllerGroup": "ipam.nephio.org", "controllerKind": "IPPrefix", "IPPrefix": {"name":"sample-n3-net-prefix1","namespace":"default"}, "namespace": "default", "name": "sample-n3-net-prefix1", "reconcileID": "bc70b264-1ead-4a4b-81cd-ae36cb0de5f1", "alloc": {"namespacedName":{"Namespace":"default","Name":"sample-n3-net-prefix1"},"origin":"prefix","networkInstance":"sample-vpc","prefixKind":"network","addressFamily":"ipv4","prefix":"10.1.0.1/24","prefixLength":24,"network":"sample-n3-net","labels":{"nephio.org/gateway":"true"}}}

[henderiw]

ok I know this issue and this is indeed a shortcut I took to get the poc going. Since you redeploy the new prefix it uses the pool as an aggregate iso the real aggregate and this is why this fails. Let me see how I can fix this.

[rravindran123]

I had left it overnight, and when i saw this evening, the conditions had resolved didnt have to delete the and reapply the n3-net-prefix1 , not sure that the controller just takes some time to inject the prefixes, also see IpPrefix/n3 in the false state..
nephio@nephio-one-summit:~$ k get ipprefixes.ipam.nephio.org
NAME SYNC STATUS NETWORK KIND NETWORK PREFIX-REQ PREFIX-ALLOC AGE
aggregate0 True True sample-vpc aggregate 10.0.0.0/8 10.0.0.0/8 37d
sample-n3-net-prefix1 True False sample-vpc network sample-n3-net 10.1.0.1/24 31h
sample-n4-net-prefix1 True True sample-vpc network sample-n4-net 192.168.1.1/24 192.168.1.1/24 37d
sample-n6-net-prefix1 True True sample-vpc network sample-n6-net 172.0.0.1/24 172.0.0.1/24 37d

[henderiw]

I was looking at this in more detail this morning and create/delete sample-n3-net-prefix1 is not a good idea, since this allows the pool to take its place since it is dynamically allocated and takes whatever prefix that is available.
The easiest way to resolve it is to split the subnet and use e.g. a different prefix for sample n3 which will never overlap with the pool.

so in nephio/config/templates/ipam.j2

change to this.

apiVersion: ipam.nephio.org/v1alpha1
kind: IPPrefix
metadata:
name: sample-n3-net-prefix1
labels:
nephio.org/gateway: "true"
spec:
prefix: 100.0.0.1/24
network: sample-n3-net
networkInstance: sample-vpc

Could you try this?

[rravindran123]

I see even just deleting and re-applying the new IPprefix works in the draft stage, but can try the above before i even clone the package to the edge deployment and see if that resolves the condition correctly

[rravindran123]

Now having done this few times, and after realizing that the delay in resolving the IP Injection condition is random, i'm assuming it is some problem in how the timers are expiring in the code and reconciling the state. Otherwise, if one has that in mind, the conditions are being resolved. But the problem of Gateway IP missing still exists.

[henderiw]

The gateway I did not analyse in detail yet but this one should be fixed with the change I proposed. Can you confirm that?

[rravindran123]

You mean the "Split IP" to handle the ip injection conditions ?, i see this is OK now..