cloud-config: honor cloud_dir setting

[aciba90]

Proposed Commit Message

cloud-config: honor cloud_dir setting

- Modules affected: cmd.main, apport, devel.logs (collect-logs),
cc_snap, sources.DataSourceAzure, sources.DataSourceBigstep,
util:fetch_ssl_details.
- testing: Extend and port to pytest unit tests, add integration test.

LP: #1976564

Additional Context

https://bugs.launchpad.net/cloud-init/+bug/1976564
https://warthogs.atlassian.net/browse/SC-1085

Test Steps

cat > /tmp/95-custom-cloud-dir.cfg <<EOF
system_info:
  paths:
    cloud_dir: /new-cloud-dir
EOF

CONTAINER_NAME=dev-k
lxc launch ubuntu-daily:kinetic $CONTAINER_NAME
lxc file push /tmp/95-custom-cloud-dir.cfg $CONTAINER_NAME/etc/cloud/cloud.cfg.d/
lxc exec $CONTAINER_NAME -- rm -rf /var/lib/cloud
lxc exec $CONTAINER_NAME -- cloud-init clean --logs --reboot
lxc exec $CONTAINER_NAME -- cloud-init status --wait --long

# Expect no logs referencing use of /var/lib/cloud
lxc exec $CONTAINER_NAME -- grep /var/lib /var/log/cloud-init.log

# Expect logs referencing use of new cloud-dir
lxc exec $CONTAINER_NAME -- grep /new-cloud-dir var/log/cloud-init.log

# Expect to find no files in /var/lib/cloud
lxc exec $CONTAINER_NAME find /var/lib/cloud

# Expect user-data file pointing to the correct cloud-dir
lxc exec $CONTAINER_NAME -- cloud-init collect-logs -h

Checklist:

• My code follows the process laid out in the documentation
• I have updated or added any unit tests accordingly
• I have updated or added any documentation accordingly

[TheRealFalcon]

@aciba90 , I noticed that this PR is in draft state. Is it still in progress, or is it ready for review?

[aciba90]

@aciba90 , I noticed that this PR is in draft state. Is it still in progress, or is it ready for review?

Yes, it is, @TheRealFalcon. I want to run the new integration test on Azure. I will let you know when ready, thanks.

[aciba90]

I have tested the added integration test on Azure, and added the Azure mark to execute it there too. This PR is ready to be reviewed @TheRealFalcon, thanks.

[TheRealFalcon]

Great work here! I left a few comments inline, but nothing major.

[aciba90]

Great work here! I left a few comments inline, but nothing major.

Thanks a lot, @TheRealFalcon! I have addressed your comments / suggestions and this PR is ready to be reviewed after the build passes.

[TheRealFalcon]

Thanks!