Remove stateless re-execution work-around

[valentindavid]

This depends on #68 #67 #66 #64 #61 #58. It will be put as ready for review as soon as all the other PRs are merged.

[alfonsosanchezbeato]

LGTM, thanks

[anonymouse64]

LGTM, thank you for splitting all this work up and submitting individually, can we update one of your snapd PR's to reference this branch (or set of commits) to see all the snapd tests run against this version of core-initrd?

[valentindavid]

LGTM, thank you for splitting all this work up and submitting individually, can we update one of your snapd PR's to reference this branch (or set of commits) to see all the snapd tests run against this version of core-initrd?

The main branch of core-initrd has since been updated for core22. Can we run snapd tests on core22?

[xnox]

we have daily edge uc22 images and testing; i think we are good to upload this into ppa, and respin kernel snaps, and then we will get full test results the next day across the uc22 testing.

[anonymouse64]

Is this change destined to only go into UC22 kernel snaps then? Or will it also be back-ported to UC20 kernel snaps?

ATM, I don't think we have any spread tests running against UC22 in snapd proper

[alfonsosanchezbeato]

Is this change destined to only go into UC22 kernel snaps then? Or will it also be back-ported to UC20 kernel snaps?

UC22 for the moment, we can backport when we think things look safe.

[anonymouse64]

Hmm, just not sure how you can declare that it looks safe given that the widest set of tests for snaps on ubuntu core is in the snapd spread suite which doesn't currently test uc22...

[alfonsosanchezbeato]

Hmm, just not sure how you can declare that it looks safe given that the widest set of tests for snaps on ubuntu core is in the snapd spread suite which doesn't currently test uc22...

I wrote "we can backport when we think things look safe", not that it looks safe...

[xnox]

Hmm, just not sure how you can declare that it looks safe given that the widest set of tests for snaps on ubuntu core is in the snapd spread suite which doesn't currently test uc22...

There are edge images for uc22 available now, is there a way to open an issue to request snapd spread suite to enable testing on uc22?

[anonymouse64]

There are edge images for uc22 available now, is there a way to open an issue to request snapd spread suite to enable testing on uc22?

One issue I just found today testing something else is that we can no longer inject snap-bootstrap as built by the snapd snap into the initramfs of the uc22 kernel snap, because it depends on libpthread.so.0:

[    1.164972] the-tool[211]: /usr/lib/snapd/snap-bootstrap: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory

Fixing this would be nice and make it easier to enable spread tests inside snapd proper

[valentindavid]

@anonymouse64 Has this been added recently? I suppose this is be added to the initrd once that version of snapd is released. Until then... Does it load libpthread.so.0 with dlopen or with a NEEDED in the dynamic section of the elf? In the former case, it has to be copied manually. In the latter, you can use dracut-install --ldd to copy binaries with all its dependencies.

[anonymouse64]

@valentindavid this happens when you just take the snap-bootstrap binary out of the snapd snap and copy it into the initramfs.

It's dynamically linked to libpthread.so.0, and indeed it's in the NEEDED dynamic section:

$ objdump -x /snap/snapd/current/usr/lib/snapd/snap-bootstrap

...

Dynamic Section:
  NEEDED               libpthread.so.0
  NEEDED               libc.so.6

[anonymouse64]

I suppose a fair point is that for these sort of tests we could just copy libpthread.so.0 into the initramfs too

[xnox]

@anonymouse64 this is maybe because you are building snap-bootstrap incorrectly in xenial? where as snap-bootstrap in core20-initrd must be built in focal; and core22-initrd must be built in jammy? since glibc 2.34 pthread functionality has moved into libc6 itself and thus builds in jammy against jammy's glibc no longer produce libpthread dependency, nor include pthread.so into the initrd, as it is now an empty library without any functionality. It is also small, because it is empty.

[xnox]

i.e. snap-bootstrap vendoring into core22-initrd should be done by extracting snap-bootstrap build from jammy.deb build; not from xenial.snap build.

[anonymouse64]

I see your point, but having to build snap-bootstrap independently from building the rest of the snapd snap to test things / do development on UC22 adds quite a bit of friction. I would be fine if it just works to copy in the libpthread.so.0 library from the snapd snap as well, I haven't tested that yet, hopefully that works. Otherwise perhaps we could somehow build snap-bootstrap to not depend on libpthread.so.0

[xnox]

we could somehow build snap-bootstrap to not depend on libpthread.so.0

if you do that, it would fail to work in core20-initrd. Building against glibc 2.33 and older generates pthread linking; and actually need it at runtime. Builds against glibc 2.34+ do not generate pthread linking, and require glibc 2.34 or newer at runtime, and will fail to run against glibc 2.33.

building snap-bootstrap alone, with its unittests, is very quick. We don't need the rest of the snapd build, to get snap-bootstrap.

[xnox]

copying pthreads.so from your build is a bad idea; you really need the empty pthreads library if you want to run against newer glibc. As you will encounter symbols clash. Do not copy old xenial pthreads into core22-initrd.

[xnox]

i am actually failing to see how xenial builds of snap-bootstrap are at all valid to inject into and test core20-initrd/core22-initrd. Since core20-initrd/core22-initrd never use such builds in production. It is interesting to know that such a combination is being tested, but it is doesn't match production at all.

[anonymouse64]

Thanks for my weekly reminder of how terrible glibc API breakage is :-(

building snap-bootstrap alone, with its unittests, is very quick. We don't need the rest of the snapd build, to get snap-bootstrap.

The most common development scenario for me as a snapd developer is usually not where I need to just test snap-bootstrap by itself, it's when I make changes to snapd codebase overall such that I need to test both userspace snapd snap changes as well as initramfs changes to snap-bootstrap, now instead of just spitting out a single snapcraft build of the snapd snap, I need to go fire up a jammy instance and build snap-bootstrap specifically there which adds to the number of steps I have to take to test things out and the number of build environments I need to maintain and use regularly.

[xnox]

Your usecase is not too dissimilar to somebody developing customized TPM hook binaries and including them in their builds. Which in similar fashion may want to build one binary and use it on both UC20 and UC22. Looking at /lib/x86_64-linux-gnu/libpthread.so.0 it is very small only 22kb and it does basically just exports version symbols only. It probably does make usability wise to just have it included.

snapd automated non-human CI, should have provisions to correctly build matching target userspace snap-bootstrap though.

And yes glibc improve/break API all the time, they limit their ABI stability only against old compiled binaries which sucks a lot. I.e. I wish on a new 2.34 glibc install I could say "compile it for at most 2.31" and for example link pthreads and only use 2.31 symbols, but alas it is not available. Despite most current 2.34 glibc having all the symbols available and exported of older glibc versions.