Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Path of open-vm-tools libdeployPkgPlugin.so is now multi-arch compliant breaking cloud-init #3909

Closed
ubuntu-server-builder opened this issue May 12, 2023 · 49 comments
Closed

Path of open-vm-tools libdeployPkgPlugin.so is now multi-arch compliant breaking cloud-init #3909

ubuntu-server-builder opened this issue May 12, 2023 · 49 comments
Labels
launchpad Migrated from Launchpad

Comments

@ubuntu-server-builder
Copy link
Collaborator

This bug was originally filed in Launchpad as LP: #1944946

Launchpad details 
affected_projects = ['cloud-init (Ubuntu)', 'open-vm-tools (Ubuntu)', 'cloud-init (Ubuntu Focal)', 'open-vm-tools (Ubuntu Focal)', 'cloud-init (Ubuntu Hirsute)', 'open-vm-tools (Ubuntu Hirsute)']
assignee = None
assignee_name = None
date_closed = 2021-11-02T19:54:26.457784+00:00
date_created = 2021-09-24T09:14:29.759294+00:00
date_fix_committed = 2021-10-12T03:55:51.147562+00:00
date_fix_released = 2021-11-02T19:54:26.457784+00:00
id = 1944946
importance = undecided
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1944946
milestone = None
owner = falcojr
owner_name = James Falcon
private = False
status = fix_released
submitter = yhzou
submitter_name = vmware-gos-Yuhua
tags = ['regression-update', 'server-next', 'verification-done-focal', 'verification-done-hirsute', 'verification-needed']
duplicates = []

Launchpad user vmware-gos-Yuhua(yhzou) wrote on 2021-09-24T09:14:29.759294+00:00

[Impact]

 * The package correctly fixed a non multiarch path, but we missed
   that some depending SW might have mad assumptions on the old paths.
   One such SW is cloud-init which in certain cases now fails to detect
   and configure vmware correctly.

 * In the long run (next Debian and 22.04) we will keep only the new
   paths. But for 21.10 time is too short and even more so for the SRUs
   that we regularly do back to at least the last LTS.
   There we want to mitigate the impact by adding a compat link on the
   old path.

[Test Plan]

 * We need to configure cloud-init to check for VMware IVMF data and we
   will see that without the fix it is failing to be detected.

 * Set up Ubuntu in VMWare if you do not ahve any ESXi then VMWare Workstation player 16 for Ubuntu as trial from https://www.vmware.com/products/workstation-player/workstation-player-evaluation.html is enough.

Reduce the ouput a bit for readability and make it not skip vmware

$ echo "datasource_list: [NoCloud, OVF]" < sudo tee /etc/cloud/cloud.cfg.d/99_test.cfg
$ echo "disable_vmware_customization: false" | sudo tee -a /etc/cloud/cloud.cfg.d/99_test.cfg

Run ds-dentify with debug on

$ sudo DEBUG_LEVEL=5 DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force

Check the result

$ cat /run/cloud-init/cloud.cfg

Bad case example:
...
Checking for datasource 'OVF' via 'dscheck_OVF'
Running on vmware but rpctool query returned 1: No value found
check for 'OVF' returned not-found[1]
found= maybe=
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 3554.80s] returning 1

And the result is:
$ cat /run/cloud-init/cloud.cfg
di_report:
  datasource_list: [ ]
  # reporting not found result. notfound=disabled.

Good case example:
...
Checking for datasource 'OVF' via 'dscheck_OVF'
Running on vmware but rpctool query returned 1: No value found
/etc/cloud/cloud.cfg.d/99_test.cfg set disable_vmware_customization to false
check for 'OVF' returned found
found=OVF maybe=
Found single datasource: OVF
[up 3357.93s] returning 0

And the result is:
$ cat /run/cloud-init/cloud.cfg
datasource_list: [ OVF, None ]

Note: VMware who spotted this will do a verification as well on this case.

[Where problems could occur]

 * Since we do not remove, but add a link (that exactly matches the
   formerly used path) I'm not too concerned. The issue I can think of
   would be e.g. security policies that prevent .so files to load through
   symlinks or anything like that. But in that case still the upload
   would not further degrade things, it would just not fix it.
   Test wise this is all about guest customizations and VMware plugin and
   VMware as usual will do checks for that when this is in verification.

[Other Info]

 * Down the road we still want to drop that path, it is only added now to
   temporarily mitigate such issues. Therefore we do NOT want to have that
   in 22.04 for a long time, and will most likely drop it there soon to
   spot further issues with it.
 * For the same reason I'm also not uploading it to Debian via
   https://salsa.debian.org/vmware-packaging-team/pkg-open-vm-tools/-/merge_requests/11 there the active release isn't affected yet
and the next one shall go without (like 22.04)

  • But for active Ubuntu release which got the backport I'd want to
    SRU fix it despite also having the cloud-init fix later on, since we do
    not know which other SW might rely on that path.

Problem:
the path of plugin libdeployPkgPlugin.so of open-vm-tools is changed and guest customization will fail for ubuntu 21.10 beta image

Description:
Ubuntu 21.10 have new open-vm-tools 11.3.0. with this new open-vm-tools, the plugin libdeployPkgPlugin.so is put to directory /usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/.

In previous open-vm-tools version (such as 11.2.5), the the plugin libdeployPkgPlugin.so is put to directory /usr/lib/open-vm-tools/plugins/vmsvc/

The path change of plugin libdeployPkgPlugin.so will cause the guest customization failure.
@ubuntu-server-builder ubuntu-server-builder added the launchpad Migrated from Launchpad label May 12, 2023
@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user vmware-gos-Yuhua(yhzou) wrote on 2021-09-24T09:22:51.509011+00:00

cloud-init 21.3-1 will search plugin libdeployPkgPlugin.so in the four paths:
search_paths = (
"/usr/lib/vmware-tools", "/usr/lib64/vmware-tools",
"/usr/lib/open-vm-tools", "/usr/lib64/open-vm-tools")

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Launchpad Janitor(janitor) wrote on 2021-09-24T09:31:38.767507+00:00

Status changed to 'Confirmed' because the bug affects multiple users.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user John Wolfe(johnwvmw) wrote on 2021-09-24T15:04:18.105356+00:00

Was the move of the open-vm-tools plugins from /usr/lib/open-vm-tools to /usr/lib/x86_64-linux-gnu/open-vm-tools an intentional change in the 11.3.0 packaging?

Would it be feasible to repackage 11.3.0, adding a symbolic link from /usr/lib/open-vm-tools to x86_64-linux-gnu/open-vm-tools ?

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-09-24T15:32:17.988937+00:00

/usr/lib/x86_64-linux-gnu/open-vm-tools is the correct path and due to the enablement or arm64 as requested we had to take care that it uses the correct path. There it will be /usr/lib/aarch64-linux-gnu/open-vm-tools or something like that.

See:
https://wiki.debian.org/Multiarch/Implementation
https://wiki.debian.org/Multiarch/Tuples

A compat link (as it is rather late) for one release and mid term fixing anything that depends on the old might be way forward.
But please note that you yourself verified already the backport thereof, so this is not only in impish but also in all of >=20.04.

I wonder if we better fix any consuming programs, is it just cloud-init or are there more we need to adapt=

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user vmware-gos-Yuhua(yhzou) wrote on 2021-09-25T00:44:31.382711+00:00

The plugin libdeployPkgPlugin.so is used for guest customization and consumed by cloud-init. I think it is just cloud-init we need to adapt.

(Note: this changed path don't affect the guest customization with perl engine in vsphere)

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user vmware-gos-Yuhua(yhzou) wrote on 2021-09-25T01:19:50.228450+00:00

John Wolfe (johnwvmw) wrote 10 hours ago:#3
Was the move of the open-vm-tools plugins from /usr/lib/open-vm-tools to /usr/lib/x86_64-linux->. gnu/open-vm-tools an intentional change in the 11.3.0 packaging?

Would it be feasible to repackage 11.3.0, adding a symbolic link from /usr/lib/open-vm-tools to >. x86_64-linux-gnu/open-vm-tools ?

test it and it works well when add a symbolic link from /usr/lib/open-vm-tools to /usr/lib/x86_64-linux-gnu/open-vm-tools

Thanks
Yuhua Zou

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-09-27T06:36:10.692820+00:00

Ok, let us retain the old path with a compat link "for now" meaning 21.10 impish and then related backports. I'm also interested what Bzed thinks about it and will throw him a PR to comment on once I have implemented this.

But at the same time we need to spawn a cloud-init bug tasks on this to be able to work with the correct future paths for 22.04 at least.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-09-27T07:08:40.807184+00:00

For clarification @john.
It isn't only .so files and not only one.
Of the former non-arch .so files we now have the following in proper directories:

open-vm-tools-desktop
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmusr/libdesktopEvents.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmusr/libdndcp.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmusr/libresolutionSet.so

open-vm-tools-dev:
./usr/lib/x86_64-linux-gnu/libDeployPkg.a
./usr/lib/x86_64-linux-gnu/libDeployPkg.so -> libDeployPkg.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestStoreClient.a
./usr/lib/x86_64-linux-gnu/libguestStoreClient.so -> libguestStoreClient.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestlib.a
./usr/lib/x86_64-linux-gnu/libguestlib.so -> libguestlib.so.0.0.0
./usr/lib/x86_64-linux-gnu/libhgfs.a
./usr/lib/x86_64-linux-gnu/libhgfs.so -> libhgfs.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvgauth.a
./usr/lib/x86_64-linux-gnu/libvgauth.so -> libvgauth.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvmtools.a
./usr/lib/x86_64-linux-gnu/libvmtools.so -> libvmtools.so.0.0.0
./usr/lib/x86_64-linux-gnu/pkgconfig/
./usr/lib/x86_64-linux-gnu/pkgconfig/libDeployPkg.pc
./usr/lib/x86_64-linux-gnu/pkgconfig/vmguestlib.pc

open-vm-tools-sdmp:
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libgdp.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libguestStore.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libserviceDiscovery.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/get-connection-info.sh
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/get-listening-process-info.sh
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/get-listening-process-perf-metrics.sh
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/get-versions.sh

open-vm-tools:
./usr/lib/x86_64-linux-gnu/libDeployPkg.so.0 -> libDeployPkg.so.0.0.0
./usr/lib/x86_64-linux-gnu/libDeployPkg.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestStoreClient.so.0 -> libguestStoreClient.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestStoreClient.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestlib.so.0 -> libguestlib.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestlib.so.0.0.0
./usr/lib/x86_64-linux-gnu/libhgfs.so.0 -> libhgfs.so.0.0.0
./usr/lib/x86_64-linux-gnu/libhgfs.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvgauth.so.0 -> libvgauth.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvgauth.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvmtools.so.0 -> libvmtools.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvmtools.so.0.0.0
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/common/libhgfsServer.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/common/libvix.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libappInfo.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libdeployPkgPlugin.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libguestInfo.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libpowerOps.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libresolutionKMS.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libtimeSync.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libvmbackup.so

I'd have assumed that plugins are only used internally, it seems odd to me that it uses a file from the plugin paths, but indeed that is what I see in
https://github.com/canonical/cloud-init/blob/main/tools/ds-identify#L902

My assumption so far would now be that:

  1. This is a non critical artifact of myself handling them as if there "could be binaries". But we might mid term want to move the shell scripts of open-vm-tools-sdmp to a non arch path as they do not have per-arch content. @john could you confirm that this path will never include binaries?
  2. as a temporary workaround for bad old deps we will add a compat link from the old /usr/lib/open-vm-tools to /usr/lib/$(DEB_HOST_MULTIARCH)/open-vm-tools

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user vmware-gos-Yuhua(yhzou) wrote on 2021-09-27T07:11:56.296536+00:00

Hi Christian Ehrhardt

let us retain the old path with a compat link "for now" meaning 21.10 impish and then related >backports.

Thanks very much.

But at the same time we need to spawn a cloud-init bug tasks on this to be able to work with the > correct future paths for 22.04 at least.
I will give feedback to vmware and ask to spawn a cloud-init bug tasks.

Best regards
Yuhua Zou

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-09-27T07:36:25.731215+00:00

As suggested #1 isn't easily doable as long as #2 is meant to be present (cant't be a symlink and a directory at the same time). And we can't be sure (unless confirmed) that
/usr/lib/$(DEB_HOST_MULTIARCH)/open-vm-tools/serviceDiscovery/scripts/ will never have compiled elements. So for now I'll hold #1 back to not make things even worse.

For #2 being the compat link that should (tm) be trivial I hope.

PR for open-vm-tools at
https://salsa.debian.org/vmware-packaging-team/pkg-open-vm-tools/-/merge_requests/11

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-09-27T09:00:52.349180+00:00

Hi John and Yuhua,
I have prepared a test build based on the change that I proved for Bernd in Salsa.
Could you have a test if this resolves (for now) the issues you have found?

=> https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4673

From the test build log:
lrwxrwxrwx root/root 0 2021-09-27 07:37 ./usr/lib/open-vm-tools -> x86_64-linux-gnu/open-vm-tools

Upgrade-installs fine over the former version.

Installed it looks like:
root@i:# ll /usr/lib/open-vm-tools
lrwxrwxrwx 1 root root 30 Sep 27 07:37 /usr/lib/open-vm-tools -> x86_64-linux-gnu/open-vm-tools/
root@i:# dpkg -S /usr/lib/open-vm-tools
open-vm-tools: /usr/lib/open-vm-tools

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user vmware-gos-Yuhua(yhzou) wrote on 2021-09-27T09:44:57.283970+00:00

Hi Christian Ehrhardt,
Sure. Thanks very much.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user vmware-gos-Yuhua(yhzou) wrote on 2021-09-27T11:06:00.755771+00:00

Hi Christian Ehrhardt,

https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4673
With this ppa it fix the issue "The path change of plugin libdeployPkgPlugin.so will cause the guest customization failure".

test with images:

  1. ubuntu 21.10 desktop beta image
  2. ubuntu 21.10 live server beta image

Check items:

  1. install open-vm-tools 11.3.0 from ppa
  2. upgrade to open-vm-tools 11.3.0 from ppa
  3. install open-vm-tools-desktop 11.3.0 from ppa for ubuntu xx.xx desktop image
  4. tools service and VGAuth service are running when install / upgrade open-vm-tools
  5. tools service and VGAuth service are running after install / upgrade and reboot guestOS
    uninstall open-vm-tools
  6. check guestInfo when install/uninstall open-vm-tools
  7. check guest customization with cloud-init engine
  8. check guest customization with perl engine

Thanks
Yuhua Zou

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user vmware-gos-Yuhua(yhzou) wrote on 2021-09-27T11:07:56.465729+00:00

Hi John Wolfe,
in #13, Any more testcases are needed due to #8 ? Thanks

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user John Wolfe(johnwvmw) wrote on 2021-09-27T13:54:56.052340+00:00

Hi Yuhua,

Please confirm that guest customization using cloud-init does, in fact, work with with this ppa.

Also, while the SDMP plugin is locating the scripts based on the installation directory, it would not hurt to run any basic SDMP test(s) available to confirm that there are no issues associated with the Muliarch packaging changes.

I am checking with the SDMP team that the script directory path has not been used as an absolute path in other applications.

Thanks Yuhua,
John

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user vmware-gos-Yuhua(yhzou) wrote on 2021-09-27T14:04:36.790295+00:00

Hi John Wolfe

Please confirm that guest customization using cloud-init does, in fact, work with with this ppa.

Yes. guest customization using cloud-init works well with ppa https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4673

I am checking with the SDMP team that the script directory path has not been used as an absolute path in other applications.

OK. Thanks.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-09-28T06:15:31.414412+00:00

Thanks everyone, from the test build and PR review POV all is fine.

I'm waiting for John to confirm that SDMP is really happy, to avoid fixing this overall issue twice.

To be clear @john - I'm ok to move the SDMP scripts subdir to a non multiarch directory if you could confirm that this will never contain architecture specific files (built code = bad, arch neutral scripts are fine). So in your talks with the SDMP team you might clarify that and depending on the outcome we might then move:
/usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/
back to
/usr/lib/open-vm-tools/serviceDiscovery/scripts/

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user John Wolfe(johnwvmw) wrote on 2021-09-28T16:14:05.383494+00:00

Hi Christian,

We are certain that the SDMP plugin ./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libserviceDiscovery.so can be found and loaded since all the other vmsvc and vmusr plugins are successfully handled.

We are waiting on the vROps team

  • to run some smoke test(s) that use the libserviceDiscovery.so
  • to confirm that serviceDiscovery/scripts/* absolute paths have not escape into other applications as happened in the cloud-init case.

If there is an escape, the patch for the symbolic link

 /usr/lib/open-vm-tools ==> /usr/lib/<arch-rt>/open-vm-tools

would compensate for any path escape while that escape is fixed down stream.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Xiaofeng Wang(xiaofengw) wrote on 2021-09-29T06:59:46.720374+00:00

Hi Christian,
I will change cloud-init codes to search libdeployPkgPlugin.so in /usr/lib//open-vm-tools directory and fix this issue. Do you know how to get the multiarch name (normalized GNU triplets) on Ubuntu and Debian? Is there a easy way to achieve this? Thanks.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-09-29T09:05:55.109078+00:00

@xiaofeng the following command will report the current one matching your machine.
$ dpkg-architecture --query DEB_HOST_MULTIARCH
If you run it without arguments you will see that there are more/other variables in case you need them.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-09-29T09:10:33.798991+00:00

@john
Thanks for the further checks, once you and the vROps team are done let me know If I shall continue as proposed in the PPA (just the compat link) or if we should generally also move serviceDiscovery/scripts/* out of the multiarch dir.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user John Wolfe(johnwvmw) wrote on 2021-09-29T18:43:31.013996+00:00

@christian

The SDMP team has confirmed that the only use of the serviceDiscovery/scripts path and contents is by the service discovery plugin. That base of that path used by the SDMP compilation and make install of the SDMP scripts is derived from $(libdir) which in turn is derived from the configure script results. There is no problem locating the scripts in the existing multi-arch library location.

They have also confirmed that the scripts path has not "escaped" outside of the open-vm-tools plugin. There will be no problem as was seen with cloud-init.

Still waiting for the vROps team smoke test results, but given the above I do not think that there will be any problems. I will let you know as soon as I hear anything.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-09-30T09:37:46.800896+00:00

Thank you - waiting for the vROps team reply then.
But given the feedback so far I think we are looking forward to upload the fix as proposed and in the PPA to Debian and Ubuntu.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-10-05T13:42:49.253447+00:00

Still waiting for a reply, considering how important this might be for some users I wanted to ping again for a response to be able to finalize this as proposed or to adapt based on the feedback.
@john - do you have any updates?

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user John Wolfe(johnwvmw) wrote on 2021-10-05T14:26:32.513409+00:00

I share your impatience. I have pinged the vROps team that is running the SDMP smoke test.

John Savanyo has been cc'ed on that e-mail.

I agree with your assessment that the fix as proposed will work. It would be nice for the confirmation.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-10-07T06:00:50.965932+00:00

Since we will slip final freeze of 21.10 not yet having that confirmation we should start preparing the SRU template which we will need later on anyway. Just to pre-eliminate any later stalls.

For that the biggest missing piece for me is the "how to reproduce" you said it affects customization via cloud-init on vmware guests.
It wasn't stated how a setup for repro for would look like yet.

I currently lack a system to test this, so I've drafted steps based on the code but would like to ask you @john to revise them. Mission - as always - is to keep them as simple as possible which e.g. implies to - if possible - not use advanced VMWare host configurations.

But from studying the code I see a few things that make me wonder which exact config is needed to reach the code that is affected by this. Here from ds-identify code.

991 dscheck_OVF() {
992 check_seed_dir ovf ovf-env.xml && return "${DS_FOUND}"

If an image was built placing data in /var/lib/cloud/seed/ovf/...
That should not happen in a fresh clear environment and will likely be stepped over.

994 [ "${DI_VIRT}" = "none" ] && return ${DS_NOT_FOUND}

This is only an early exit, if it is vmware it continues to check the rest

996 # Azure provides ovf. Skip false positive by dis-allowing.
997 is_azure_chassis && return $DS_NOT_FOUND

Not important in our cases as we are not talking about Azure, will be stepped over

999 ovf_vmware_transport_guestinfo && return "${DS_FOUND}"

If here vmware-rpctool "info-get guestinfo.ovfEnv" gets content it returns with DS_FOUND

1001 has_ovf_cdrom && return "${DS_FOUND}"

If cloud-init config data is presented via cdrom it returns here.

1003 ovf_vmware_guest_customization && return "${DS_FOUND}"

Only if it stepped over all of the above, no config CDrom, no data via vmware-rpctool then the code of "ovf_vmware_guest_customization" which has the issue with the new path will be ran.

1005 return ${DS_NOT_FOUND}

This is the final, nothing found return

WIP Steps to reproduce to improve on

  1. set up a VMWare based Ubuntu guest
    1.1 does the user or the tooling have to set disable_vmware_customization ?
    something like
    echo "search,found=all,maybe=all,notfound=disabled,disable_vmware_customization=false" | sudo tee /etc/cloud/ds-identify.cfg

1.2 do any other options/customizations matter?

  1. in the Ubuntu guest run ds-identify and check if the output detected vmware correctly

2.1 $ sudo /usr/lib/cloud-init/ds-identify

2.2 $ cat /run/cloud-init/cloud.cfg

In the good case this looks like ???
In the bad case it fails to detect VMware and looks like ???

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-10-07T07:16:20.685710+00:00

Furthermore I tried to check this in VMWare Workstation player 16

First of all as I'd expect systemd-detect-virt identified vmware already.
$ systemd-detect-virt
vmware

The code affected by this issue here is in "ovf_vmware_guest_customization".
That is only used in dscheck_OVF.
Therefore to streamline the run reduce config to just detect OVF in ds-identify.

$ echo "datasource_list: [NoCloud, OVF]" < sudo tee /etc/cloud/cloud.cfg.d/99_test.cfg

Running ds-identify in verbose more to check behavior.

As expected I see it pass the steps before and reach ovf_vmware_guest_customization.
And indeed it fails to detect the pluging and does an early exit.

Here with a bit of additional debug inside of ovf_vmware_guest_customization:
$ sudo DEBUG_LEVEL=5 DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force
[up 2345.15s] ds-identify --force
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
/etc/cloud/cloud.cfg.d/90_dpkg.cfg set datasource_list: [NoCloud, OVF]
DMI_PRODUCT_NAME=VMware Virtual Platform
DMI_SYS_VENDOR=VMware, Inc.
DMI_PRODUCT_SERIAL=VMware-56 4d 5c 48 27 3b 66 c7-18 31 7d 99 e7 30 40 7e
DMI_PRODUCT_UUID=485c4d56-3b27-c766-1831-7d99e730407e
PID_1_PRODUCT_NAME=unavailable
DMI_CHASSIS_ASSET_TAG=No Asset Tag
FS_LABELS=
ISO9660_DEVS=
KERNEL_CMDLINE=BOOT_IMAGE=/boot/vmlinuz-5.11.0-20-generic root=UUID=e6b98230-678e-4963-88a1-673f9e943acd ro find_preseed=/preseed.cfg auto noprompt priority=critical locale=en_US quiet
VIRT=vmware
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_RELEASE=5.11.0-20-generic
UNAME_KERNEL_VERSION=#21+21.10.1-Ubuntu SMP Wed Jun 9 15:08:14 UTC 2021
UNAME_MACHINE=x86_64
UNAME_NODENAME=ubuntu
UNAME_OPERATING_SYSTEM=GNU/Linux
DSNAME=
DSLIST=NoCloud OVF
MODE=search
ON_FOUND=all
ON_MAYBE=all
ON_NOTFOUND=disabled
pid=54738 ppid=54737
is_container=false
Checking for datasource 'NoCloud' via 'dscheck_NoCloud'
check for 'NoCloud' returned not-found[1]
Checking for datasource 'OVF' via 'dscheck_OVF'
DEBUG: dscheck_OVF: Check seed
DEBUG: dscheck_OVF: DI_VIRT
DEBUG: dscheck_OVF: is_azure_chassis
DEBUG: dscheck_OVF: ovf_vmware_transport_guestinfo
Running on vmware but rpctool query returned 1: No value found
DEBUG: dscheck_OVF: has_ovf_cdrom
DEBUG: dscheck_OVF: ovf_vmware_guest_customization
DEBUG: ovf_vmware_guest_customization: start check
DEBUG: scan plugin
check for 'OVF' returned not-found[1]
found= maybe=
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 2345.23s] returning 1

With the fix (from PPA https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4673) in place it finds the plugin as expected.

We want to also force vmware customization itself for this test which isn't the default.

$ echo "disable_vmware_customization: false" | sudo tee -a /etc/cloud/cloud.cfg.d/99_test.cfg

And then the result is in /etc/cloud-init/cloud.cfg

We will see the check say:
...
Checking for datasource 'OVF' via 'dscheck_OVF'
Running on vmware but rpctool query returned 1: No value found
/etc/cloud/cloud.cfg.d/99_test.cfg set disable_vmware_customization to false
check for 'OVF' returned found
found=OVF maybe=
Found single datasource: OVF
[up 3357.93s] returning 0

And the result is:
$ cat /run/cloud-init/cloud.cfg
datasource_list: [ OVF, None ]

In comparison the bad case has:
...
Checking for datasource 'OVF' via 'dscheck_OVF'
Running on vmware but rpctool query returned 1: No value found
check for 'OVF' returned not-found[1]
found= maybe=
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 3554.80s] returning 1

And the result is:
$ cat /run/cloud-init/cloud.cfg
di_report:
datasource_list: [ ]

reporting not found result. notfound=disabled.

That is enough to construct a testcase for the SRU template ...

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-10-07T07:21:42.476820+00:00

SRU Template ready now, juts waiting for that final confirmation from vROps team to start the upload to 21.10 and the SRUs from there.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-10-07T07:27:13.486393+00:00

TBH @john - I'm tempted to do this upload to Impish now for all the inbound deadlines.
Chances are quite high that this will be the right solution and today (Oct 7th) is the final freeze for 21.10 (https://discourse.ubuntu.com/t/impish-indri-release-schedule/18540).

Since this issue is severe for deployment and open-vm-tools it is a release critical bug that needs to go in before the release is ready.

But also it is seeded, so uploading this later will force us to re-build the RC ISOs/Images and to re-start testing.

I can either
a) wait and surely break & reset the 21.10 release once we have that confirmation
or
b) upload this quite likely fix now, and only if we spot issues re-upload a modification

Please continue to push on the vrops team, but clearly (b) should in any case be better for everyone, especially if it is confirmed to be the right fix.

Therefore I'll mark the bug accordingly and do the upload ...

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-10-07T08:48:33.911919+00:00

Uploaded, accepted by the release team and built at https://launchpad.net/ubuntu/+source/open-vm-tools/2:11.3.0-2ubuntu1
Now enters automated testing and hopefully migrating before the first images/ISOs are built.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Launchpad Janitor(janitor) wrote on 2021-10-07T09:36:12.527747+00:00

This bug was fixed in the package open-vm-tools - 2:11.3.0-2ubuntu1

open-vm-tools (2:11.3.0-2ubuntu1) impish; urgency=medium

  • d/rules: provide a compat link for the old open-vm-tools
    library/plugin paths (LP: #1944946)

-- Christian Ehrhardt christian.ehrhardt@canonical.com Thu, 07 Oct 2021 09:31:05 +0200

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-10-07T11:37:53.818562+00:00

@john - ok that worked well, let us hope no later fixups are needed.
Before I go for SRU of this to the other releases with 11.3 i'm waiting for this final feedback by you and the vRops team.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user John Wolfe(johnwvmw) wrote on 2021-10-07T14:18:11.385813+00:00

@christian - Just got the test results.

The Service Discovery smoke test(s) PASSED. The multi-arch layout does not impact the SDMP plugin.

The Guest Customization team will be resolving the libdeployPkgPlugin check in the cloud-init project.

  • quickly, by adding the multi-arch paths to the list of paths to be searched for the .so
  • revise the guest customization availability check to not depend on the list of fixed paths.
    (longer range fix)

In either case going forward, the /usr/lib/open-vm-tools symbolic link should only be needed in 11.3.0 and 11.3.5 open-vm-tools. It should not be needed by OVT 12.0.0 and Ubuntu 22.04.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-10-07T14:47:15+00:00

Thanks for the confirmations

In either case going forward, the /usr/lib/open-vm-tools symbolic link
should only be needed in 11.3.0 and 11.3.5 open-vm-tools.
It should not be needed by OVT 12.0.0 and Ubuntu 22.04.

My intention was to drop it slightly earlier.
I wanted to only have it in 11.3.0, but already drop it in 11.3.5 for
next Debian and 22.04.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user John Wolfe(johnwvmw) wrote on 2021-10-07T16:23:02.934655+00:00

I was being conservative, not being aware of:

  • when the path fix will be made to cloud-init
  • when the next release of cloud-init (with the fix) will be available.

Open-vm-tools 11.3.5 on Ubuntu and Debian w/o the /usr/lib/open-vm-tools symbolic link will have a dependency on the updated next release of cloud-init.

I have raised the priority of the internal PR to guest customization has for the cloud-init fix.

Please keep and eye out for a pull request.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user James Falcon(falcojr) wrote on 2021-10-12T03:56:30.704171+00:00

cloud-init fix has been committed upstream via #1061

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Xiaofeng Wang(xiaofengw) wrote on 2021-10-12T04:27:38.824357+00:00

The cloud-init fix is delivered. So next Debian and Ububutu 22.04 will pick up this change, is it right? Is there any action I need to do for this issue? Thanks.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-10-12T05:23:42.980359+00:00

Hi Xiaofeng,
cloud-init releases regularly across Ubuntu - so even older releases will pick up the change with the next round of cloud-init updates. Unless the plan has changed that would be in a few weeks 1 plus some time to make it into SRUs from there.

But in addition I think we should also SRU the mitigation we've added to 21.10 open-vm-tools to ensure there is no other (than cloud-init) SW broken by this. I'll therefore prep these SRUs despite a fix in cloud-init coming as well.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2021-10-12T05:59:09.961601+00:00

FYI MPs and PPA uploads for Focal/Hirsute uploaded:
F: https://code.launchpad.net/~paelzer/ubuntu/+source/open-vm-tools/+git/open-vm-tools/+merge/410021
H: https://code.launchpad.net/~paelzer/ubuntu/+source/open-vm-tools/+git/open-vm-tools/+merge/410022
PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4673

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Brian Murray(brian-murray) wrote on 2021-10-19T19:09:13.805226+00:00

Hello vmware-gos-Yuhua, or anyone else affected,

Accepted open-vm-tools into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-vm-tools/2:11.3.0-2ubuntu0~ubuntu21.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Brian Murray(brian-murray) wrote on 2021-10-19T19:11:03.578714+00:00

Hello vmware-gos-Yuhua, or anyone else affected,

Accepted open-vm-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-vm-tools/2:11.3.0-2ubuntu0~ubuntu20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user vmware-gos-Yuhua(yhzou) wrote on 2021-10-23T03:07:33.753974+00:00

The issue is fixed when check it in the following images.

  1. ubuntu 20.04.3 desktop GA (-proposed repo)
  2. ubuntu 21.04 desktop GA. (-proposed repo)
  3. ubuntu 21.10 desktop GA
  4. ubuntu 21.10 live server GA
  5. ubuntu 21.10 cloud image

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Łukasz Zemczak(sil2100) wrote on 2021-10-26T13:19:42.088650+00:00

Okay, I'll release this conditionally as I see all the respective series have been probably tested, but please be sure to include more information regarding which testing has been performed and on which package versions exactly.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Launchpad Janitor(janitor) wrote on 2021-10-26T13:19:56.216957+00:00

This bug was fixed in the package open-vm-tools - 2:11.3.0-2ubuntu0~ubuntu21.04.2

open-vm-tools (2:11.3.0-2ubuntu0~ubuntu21.04.2) hirsute; urgency=medium

  • d/rules: provide a compat link for the old open-vm-tools
    library/plugin paths (LP: #1944946)
    • d/open-vm-tools.postinst: handle upgrades from <11.3.0-2 in regard
      to the symlink

-- Christian Ehrhardt christian.ehrhardt@canonical.com Tue, 12 Oct 2021 07:50:08 +0200

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Łukasz Zemczak(sil2100) wrote on 2021-10-26T13:20:01.585016+00:00

The verification of the Stable Release Update for open-vm-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Launchpad Janitor(janitor) wrote on 2021-10-26T13:34:30.579750+00:00

This bug was fixed in the package open-vm-tools - 2:11.3.0-2ubuntu0~ubuntu20.04.2

open-vm-tools (2:11.3.0-2ubuntu0~ubuntu20.04.2) focal; urgency=medium

  • d/rules: provide a compat link for the old open-vm-tools
    library/plugin paths (LP: #1944946)
    • d/open-vm-tools.postinst: handle upgrades from <11.3.0-2 in regard
      to the symlink

-- Christian Ehrhardt christian.ehrhardt@canonical.com Tue, 12 Oct 2021 07:53:54 +0200

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user James Falcon(falcojr) wrote on 2021-11-02T19:54:27.873558+00:00

This bug is believed to be fixed in cloud-init in version 21.4. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Launchpad Janitor(janitor) wrote on 2021-11-03T07:06:45.015913+00:00

This bug was fixed in the package cloud-init - 21.4-0ubuntu1~22.04.1

cloud-init (21.4-0ubuntu1~22.04.1) jammy; urgency=medium

-- James Falcon james.falcon@canonical.com Tue, 02 Nov 2021 18:07:49 -0500

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2022-01-04T11:15:30.999724+00:00

FYI - as planned (and now that cloud init is fixed) I'm planning to upload this to Jammy without the path fix. There should be no issue with that.

But planning ahead, there will later on be a backport of open-vm-tools 11.3.5 with that to >=Focal.
I checked
21.4-0ubuntu120.04.1
21.4-0ubuntu121.04.1
Both have the fix and therefor do not block the backport.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
launchpad Migrated from Launchpad
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ubuntu-server-builder