feature(schema): add networkv2 schema

[catmsred]

Adds networkv2 schema definition file, removes skipping of schema validation for networkv2, and adds unit tests for same.

Proposed Commit Message

feature(schema): add networkv2 schema
    
Adds networkv2 schema definition file, removes skipping of schema
validation for networkv2, and adds unit tests for same.

Additional Context

Having a defined schema will help track drift from netplan schema.

Checklist

• My code follows the process laid out in the documentation
• I have updated or added any unit tests accordingly
• I have updated or added any documentation accordingly

Merge type

• Squash merge using "Proposed Commit Message"
• Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

[catmsred]

Need to fix linting and unit tests still

[TheRealFalcon]

Since I'm on triage this week, I probably won't get fully reviewing this until next week or later, so I'm going to unassign myself if somebody else would like to pick it up in the mean time.

I'll leave some comments from when I started the review, but my review isn't complete.

Also, we'll need some way of wiring this schema up to the cloud-init schema command so that somebody can pass a v1 or v2 schema and have it verified. E.g., right now you can do something like
cloud-init schema --config-file my-schema.yaml -t network-config

but we'll probably need to expand -t to include v1 and v2 (or auto-detect it) and use the correct schema accordingly.

[TheRealFalcon]

The way we've named things is confusing, but this isn't really what we want here. The versioning here is for versions of the schemas themselves (which as of now and possibly forever, they aren't explicitly versioned), not the network configuration version.

Part of me is tempted to remove this section entirely, but if we keep it we should have a NETWORK_CONFIG_V1 and NETWORK_CONFIG_V2 under the SchemaType enum with separate latest versions here.

[catmsred]

Thanks to James' comments I have updated how we handle the different schema versions. I elected to leave -t to autodetect the schema version (since the code to do so was already in place) but am open to suggestions on how that path goes at the moment.

[aciba90]

Thanks, @catmsred, for working on this!

validate_cloudconfig_schema is called from:

1. cloudinit.stages to validate the network config at cloud-init runtime before applying it and
2. From validate_cloudconfig_file, which is used in the cloud-init schema cli command handler.

As it is right now, validate_cloudconfig_schema unconditionally uses netplan_validate_network_schema to validate v2 netconfigs and netplan_validate_network_schema uses Netplan machinery to validate if present, but returns False when Netplan APIs are not installed. Thus, the schema added in this PR is not used in both cases, but it could be used on such systems, and that was the intention of this effort, or am I missing something?

In addition to that, ideally, I think it might be worth adding integration tests covering both cases.

[aciba90]

Here, schema gets overwritten if given by the user as argument.

[catmsred]

Yes it does since in the places it's called within this code, schema is incorrectly set to V2 so it's important to reset it after the version has been dynamically determined from the actual config.

[aciba90]

That makes sense, thanks. Then the call-sites that pass schema_type=SchemaType.NETWORK_CONFIG and a given schema, are going to read some schema two times. I think we could probably optimize that and not pass a schema in that case.

[catmsred]

As it is right now, validate_cloudconfig_schema unconditionally uses netplan_validate_network_schema to validate v2 netconfigs and netplan_validate_network_schema uses Netplan machinery to validate if present, but returns False when Netplan APIs are not installed. Thus, the schema added in this PR is not used in both cases, but it could be used on such systems, and that was the intention of this effort, or am I missing something?

No, that is not the intention and thanks for catching that. I'll refactor to make sure that netplan_validate_network_schema is only used when netplan is present.

In addition to that, ideally, I think it might be worth adding integration tests covering both cases.

Will do!

[catmsred]

As it is right now, validate_cloudconfig_schema unconditionally uses netplan_validate_network_schema to validate v2 netconfigs and netplan_validate_network_schema uses Netplan machinery to validate if present, but returns False when Netplan APIs are not installed. Thus, the schema added in this PR is not used in both cases, but it could be used on such systems, and that was the intention of this effort, or am I missing something?

No, that is not the intention and thanks for catching that. I'll refactor to make sure that netplan_validate_network_schema is only used when netplan is present.

I went back to work on this and realized I think I'm not fully understanding your concerns. On a system with netplan, netplan_validate_network_schema will validate the schema and validate_cloudconfig_schema will return True as it should since netplan will be the program implementing the config on such a system. On a system without netplan but with networkv2, netplan_validate_network_schema will return False (since there is no netplan) and validate_cloudconfig_schema will continue running to validate the schema against the networkv2 schema included in this PR, which also seems to be correct behavior to me. Note that when validate_cloudconfig_schema calls netplan_validate_network_schema it doesn't break on return on a False from netplan_validate_network_schema.

[aciba90]

As it is right now, validate_cloudconfig_schema unconditionally uses netplan_validate_network_schema to validate v2 netconfigs and netplan_validate_network_schema uses Netplan machinery to validate if present, but returns False when Netplan APIs are not installed. Thus, the schema added in this PR is not used in both cases, but it could be used on such systems, and that was the intention of this effort, or am I missing something?

No, that is not the intention and thanks for catching that. I'll refactor to make sure that netplan_validate_network_schema is only used when netplan is present.

I went back to work on this and realized I think I'm not fully understanding your concerns. On a system with netplan, netplan_validate_network_schema will validate the schema and validate_cloudconfig_schema will return True as it should since netplan will be the program implementing the config on such a system. On a system without netplan but with networkv2, netplan_validate_network_schema will return False (since there is no netplan) and validate_cloudconfig_schema will continue running to validate the schema against the networkv2 schema included in this PR, which also seems to be correct behavior to me. Note that when validate_cloudconfig_schema calls netplan_validate_network_schema it doesn't break on return on a False from netplan_validate_network_schema.

Yeah, that's right, I read too fast that part of the code. Thanks for pointing that out.

I left some in-line comments, but this LGTM, thanks!

[aciba90]

That makes sense, thanks. Then the call-sites that pass schema_type=SchemaType.NETWORK_CONFIG and a given schema, are going to read some schema two times. I think we could probably optimize that and not pass a schema in that case.

[aciba90]

This also accepts SchemaType.NETWORK_CONFIG, do you think is worth simplifying this?

@param schema_type: Instance of SchemaType.

[TheRealFalcon]

@catmsred and @aciba90 , where are we at on this one?

[catmsred]

@catmsred and @aciba90 , where are we at on this one?

I need to write integration tests -- got distracted by other things this week. I will try to get it done soon.

[TheRealFalcon]

No problem; there's no rush on this one. I just wanted to make sure nobody was blocked or waiting for somebody else.

[aciba90]

Thanks, @catmsred, for adding integration tests. I have done a partial review and left some comments / requests.

The test with valid config does not pass, thus something is off either with the test or the implementation.

[catmsred]

Since there are no automated tests in a non-netplan environment, I did some manual testing to validate the new schema on a non-netplan distro. I used alpine, as follows:

# install cloud-init
$incus exec alpine-edge -- apk add cloud-init

# checkout this branch of cloud-init
$incus exec alpine-edge -- sh -c 'cd cloud-init && git checkout feature/networkv2schema'

# copy the schema file and validation code to running cloud-init code
$incus exec alpine-edge -- sudo cp cloud-init/cloudinit/config/schema.py /usr/lib/python3.12/site-packages/cloudinit/config/schema.py
$incus exec alpine-edge -- cp cloud-init/cloudinit/config/schemas/schema-network-config-v2.json /usr/lib/python3.12/site-packages/cloudinit/config/schemas/schema-network-config-v2.json

# create valid networkv2 schema
$incus exec alpine-edge -- sh -c 'echo -e "network:\n  ethernets:\n    eth0:\n      dhcp4: true\n  version: 2" > test_networkv2'

# validate valid networkv2 schema
$incus exec alpine-edge -- cloud-init schema --config-file test_networkv2 -t network-config
Valid schema test_networkv2

# create invalid networkv2 schema
$incus exec alpine-edge -- sh -c 'echo -e "network:\n  ethernets:\n    eth0:\n      dhcp4: bogus\n  version: 2" > test_networkv2_invalid'

# validate invalid networkv2 schema
$incus exec alpine-edge -- cloud-init schema --config-file test_networkv2_invalid -t network-config
Invalid network-config test_networkv2_invalid
Error: Cloud config schema errors: network.ethernets.eth0.dhcp4: 'bogus' is not one of ['yes', 'no', True, False]

Error: Invalid schema: network-config

[aciba90]

LGTM, thanks!