VDSO errors on ppc64le on 5.11.x #1417

adrianreber · 2021-03-24T09:36:06Z

While trying to debug #1415 I updated to a newer kernel (5.11.7-100.fc32.ppc64le) and now CRIU completely fails on ppc64le:

# criu/criu check -v4
(00.000002) Version: 3.15 (gitid v3.14-340-g480605824)
(00.000012) Running on ibm-p9z-18-lp5.virt.pnr.lab.eng.rdu2.redhat.com Linux 5.11.7-100.fc32.ppc64le #1 SMP Wed Mar 17 18:39:54 UTC 2021 ppc64le
(00.000024) File /run/criu.kdat does not exist
(00.000044) sockets: Probing sock diag modules
(00.000095) sockets: Done probing
(00.003082) Pagemap is fully functional
(00.003113) Found anon-shmem device at 1
(00.003120) Reset 1201's dirty tracking
(00.003367)  ... done
(00.003455) Dirty track supported on kernel
(00.003534) Found task size of 2000000000000
(00.007536) Restoring netdev veth idx 10
(00.007789) Dumping netns links
(00.007816) 	LD: Got link 1, type 772
(00.007824) 	LD: Got link 10, type 1
(00.008982) vdso: Parsing at 7fffb9070000 7fffb9090000
(00.008988) Error (criu/pie-util-vdso.c:97): vdso: ELF header magic mismatch
(00.008991) Error (criu/vdso.c:634): vdso: Failed to fill self vdso symtable
(00.008997) Error (criu/kerndat.c:1152): kerndat_vdso_fill_symtable failed when initializing kerndat.
(00.009096) Adjust mmap_min_addr 0x1000 -> 0x10000
(00.009101) Found mmap_min_addr 0x10000
(00.009113) files stat: fs/nr_open 1073741816
(00.009118) Error (criu/crtools.c:213): Could not initialize kernel features detection.

The error looks the same if booting with vdso=0 or without.

I added some debug code to see why it fails and read ehdr->e_ident is different from the expected.

CRIU expects 7F 45 4C 46 2 1 1 0 0 0 0 0 0 0 0 0 but CRIU actually reads 53 59 53 54 45 4D 43 46 47 3A 50 50 43 36 34 0

@0x7f454c46 maybe you have and idea. This was still working on 5.8.18-100.fc31.ppc64le

@mihalicyn just FYI if you are planning to upgrade the Jenkins hosts you might run into this error.

The text was updated successfully, but these errors were encountered:

rppt · 2021-03-24T13:41:55Z

@ldu4

ldu4 · 2021-03-25T11:33:47Z

That may be tied to the switch of the powerpc VDSO to generic C implementation:
https://lore.kernel.org/linuxppc-dev/cover.1604426550.git.christophe.leroy@csgroup.eu/

ldu4 · 2021-03-25T11:36:40Z

BTW,
53 59 53 54 45 4D 43 46 47 3A 50 50 43 36 34 0 = "SYSTEMCFG:PPC64"
and
7F 45 4C 46 2 1 1 0 0 0 0 0 0 0 0 0 = "ELF!"

chleroy · 2021-03-25T16:50:00Z

Probably linked to commit linuxppc/linux@511157a which has put the data page up front like most other architectures.

ldu4 · 2021-03-25T19:31:44Z

This can be managed by reading the aux vector AT_SYSINFO_EHDR.
I wrote a patch ldu4/criu@e870e1434648 which makes criu check happy.

@adrianreber could you give it a try?

0x7f454c46 · 2021-03-25T20:27:41Z

Hi guys, @chleroy @ldu4, what do you think about this kernel patch?
0x7f454c46/linux@783c7a2
(untested, working on it, I appreciate if someone helps with it)

0x7f454c46 · 2021-03-25T20:46:16Z

Hi @ldu4, one thing I find a bit strange about your patch is that you fixed vdso_fill_self_symtable(), but probably restorer has to be fixed too: it will read vdso vma from images, but than it will try to parse it in vdso_proxify(). Where it will try to parse ELF at the start of the VMA.

Since commit 511157a ("powerpc/vdso: Move vdso datapage up front") VVAR page is in front of the VDSO area. In result it breaks CRIU (Checkpoint Restore In Userspace) [1], where CRIU expects that "[vdso]" from /proc/../maps points at ELF/vdso image, rather than at VVAR data page. Laurent made a patch to keep CRIU working (by reading aux vector). But I think it still makes sence to separate two mappings into different VMAs. It will also make ppc64 less "special" for userspace and as a side-bonus will make VVAR page un-writable by debugger (which previously would COW page and can be unexpected). I opportunistically Cc stable on it: I understand that usually such stuff isn't a stable material, but that will allow us in CRIU have one workaround less that is needed just for one release (v5.11) on one platform (ppc64), which we otherwise have to maintain. I wouldn't go as far as to say that the commit 511157a is ABI regression as no other userspace got broken, but I'd really appreciate if it gets backported to v5.11 after v5.12 is released, so as not to complicate already non-simple CRIU-vdso code. Thanks! Cc: Andrei Vagin <avagin@gmail.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Christophe Leroy <christophe.leroy@csgroup.eu> Cc: Laurent Dufour <ldufour@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Paul Mackerras <paulus@samba.org> Cc: linuxppc-dev@lists.ozlabs.org Cc: stable@vger.kernel.org # v5.11 [1]: checkpoint-restore/criu#1417 Signed-off-by: Dmitry Safonov <dima@arista.com> Tested-by: Christophe Leroy <christophe.leroy@csgroup.eu>

ldu4 · 2021-03-29T09:41:30Z

Hi @0x7f454c46, I probably missed other part, I did that in hurry and didn't get more time to do C/R test. I'll investigate on that.
That being said, I think your kernel patch introducing the vvar mapping for PowerPC makes sense.

Since commit 511157a ("powerpc/vdso: Move vdso datapage up front") VVAR page is in front of the VDSO area. In result it breaks CRIU (Checkpoint Restore In Userspace) [1], where CRIU expects that "[vdso]" from /proc/../maps points at ELF/vdso image, rather than at VVAR data page. Laurent made a patch to keep CRIU working (by reading aux vector). But I think it still makes sence to separate two mappings into different VMAs. It will also make ppc64 less "special" for userspace and as a side-bonus will make VVAR page un-writable by debugger (which previously would COW page and can be unexpected). I opportunistically Cc stable on it: I understand that usually such stuff isn't a stable material, but that will allow us in CRIU have one workaround less that is needed just for one release (v5.11) on one platform (ppc64), which we otherwise have to maintain. I wouldn't go as far as to say that the commit 511157a is ABI regression as no other userspace got broken, but I'd really appreciate if it gets backported to v5.11 after v5.12 is released, so as not to complicate already non-simple CRIU-vdso code. Thanks! Cc: Andrei Vagin <avagin@gmail.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Christophe Leroy <christophe.leroy@csgroup.eu> Cc: Laurent Dufour <ldufour@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Paul Mackerras <paulus@samba.org> Cc: linuxppc-dev@lists.ozlabs.org Cc: stable@vger.kernel.org # v5.11 [1]: checkpoint-restore/criu#1417 Signed-off-by: Dmitry Safonov <dima@arista.com> Tested-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>

Since commit 511157a ("powerpc/vdso: Move vdso datapage up front") VVAR page is in front of the VDSO area. In result it breaks CRIU (Checkpoint Restore In Userspace) [1], where CRIU expects that "[vdso]" from /proc/../maps points at ELF/vdso image, rather than at VVAR data page. Laurent made a patch to keep CRIU working (by reading aux vector). But I think it still makes sence to separate two mappings into different VMAs. It will also make ppc64 less "special" for userspace and as a side-bonus will make VVAR page un-writable by debugger (which previously would COW page and can be unexpected). I opportunistically Cc stable on it: I understand that usually such stuff isn't a stable material, but that will allow us in CRIU have one workaround less that is needed just for one release (v5.11) on one platform (ppc64), which we otherwise have to maintain. I wouldn't go as far as to say that the commit 511157a is ABI regression as no other userspace got broken, but I'd really appreciate if it gets backported to v5.11 after v5.12 is released, so as not to complicate already non-simple CRIU-vdso code. Thanks! [1]: checkpoint-restore/criu#1417 Cc: stable@vger.kernel.org # v5.11 Signed-off-by: Dmitry Safonov <dima@arista.com> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Link: https://lore.kernel.org/r/f401eb1ebc0bfc4d8f0e10dc8e525fd409eb68e2.1617209142.git.christophe.leroy@csgroup.eu

Since commit 511157a ("powerpc/vdso: Move vdso datapage up front") VVAR page is in front of the VDSO area. In result it breaks CRIU (Checkpoint Restore In Userspace) [1], where CRIU expects that "[vdso]" from /proc/../maps points at ELF/vdso image, rather than at VVAR data page. Laurent made a patch to keep CRIU working (by reading aux vector). But I think it still makes sence to separate two mappings into different VMAs. It will also make ppc64 less "special" for userspace and as a side-bonus will make VVAR page un-writable by debugger (which previously would COW page and can be unexpected). I opportunistically Cc stable on it: I understand that usually such stuff isn't a stable material, but that will allow us in CRIU have one workaround less that is needed just for one release (v5.11) on one platform (ppc64), which we otherwise have to maintain. I wouldn't go as far as to say that the commit 511157a is ABI regression as no other userspace got broken, but I'd really appreciate if it gets backported to v5.11 after v5.12 is released, so as not to complicate already non-simple CRIU-vdso code. Thanks! [1]: checkpoint-restore/criu#1417 Cc: stable@vger.kernel.org # v5.11 Signed-off-by: Dmitry Safonov <dima@arista.com> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Christophe Leroy <christophe.leroy@csgroup.eu> Acked-by: Andrei Vagin <avagin@gmail.com> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Link: https://lore.kernel.org/r/f401eb1ebc0bfc4d8f0e10dc8e525fd409eb68e2.1617209142.git.christophe.leroy@csgroup.eu

Since commit 511157a ("powerpc/vdso: Move vdso datapage up front") VVAR page is in front of the VDSO area. In result it breaks CRIU (Checkpoint Restore In Userspace) [1], where CRIU expects that "[vdso]" from /proc/../maps points at ELF/vdso image, rather than at VVAR data page. Laurent made a patch to keep CRIU working (by reading aux vector). But I think it still makes sence to separate two mappings into different VMAs. It will also make ppc64 less "special" for userspace and as a side-bonus will make VVAR page un-writable by debugger (which previously would COW page and can be unexpected). I opportunistically Cc stable on it: I understand that usually such stuff isn't a stable material, but that will allow us in CRIU have one workaround less that is needed just for one release (v5.11) on one platform (ppc64), which we otherwise have to maintain. I wouldn't go as far as to say that the commit 511157a is ABI regression as no other userspace got broken, but I'd really appreciate if it gets backported to v5.11 after v5.12 is released, so as not to complicate already non-simple CRIU-vdso code. Thanks! [1]: checkpoint-restore/criu#1417 Cc: stable@vger.kernel.org # v5.11 Signed-off-by: Dmitry Safonov <dima@arista.com> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Christophe Leroy <christophe.leroy@csgroup.eu> Reviewed-by: Vincenzo Frascino <vincenzo.frascino@arm.com> # vDSO parts. Acked-by: Andrei Vagin <avagin@gmail.com> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Link: https://lore.kernel.org/r/f401eb1ebc0bfc4d8f0e10dc8e525fd409eb68e2.1617209142.git.christophe.leroy@csgroup.eu

github-actions · 2021-04-29T00:06:41Z

A friendly reminder that this issue had no activity for 30 days.

commit 1c4bce6 upstream. Since commit 511157a ("powerpc/vdso: Move vdso datapage up front") VVAR page is in front of the VDSO area. In result it breaks CRIU (Checkpoint Restore In Userspace) [1], where CRIU expects that "[vdso]" from /proc/../maps points at ELF/vdso image, rather than at VVAR data page. Laurent made a patch to keep CRIU working (by reading aux vector). But I think it still makes sence to separate two mappings into different VMAs. It will also make ppc64 less "special" for userspace and as a side-bonus will make VVAR page un-writable by debugger (which previously would COW page and can be unexpected). I opportunistically Cc stable on it: I understand that usually such stuff isn't a stable material, but that will allow us in CRIU have one workaround less that is needed just for one release (v5.11) on one platform (ppc64), which we otherwise have to maintain. I wouldn't go as far as to say that the commit 511157a is ABI regression as no other userspace got broken, but I'd really appreciate if it gets backported to v5.11 after v5.12 is released, so as not to complicate already non-simple CRIU-vdso code. Thanks! [1]: checkpoint-restore/criu#1417 Cc: stable@vger.kernel.org # v5.11 Signed-off-by: Dmitry Safonov <dima@arista.com> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Christophe Leroy <christophe.leroy@csgroup.eu> Reviewed-by: Vincenzo Frascino <vincenzo.frascino@arm.com> # vDSO parts. Acked-by: Andrei Vagin <avagin@gmail.com> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Link: https://lore.kernel.org/r/f401eb1ebc0bfc4d8f0e10dc8e525fd409eb68e2.1617209142.git.christophe.leroy@csgroup.eu Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

adrianreber · 2021-05-12T16:13:05Z

Looks like the kernel changes made it to 5.13, 5.11.20 and 5.12.3

commit 1c4bce6 upstream. Since commit 511157a ("powerpc/vdso: Move vdso datapage up front") VVAR page is in front of the VDSO area. In result it breaks CRIU (Checkpoint Restore In Userspace) [1], where CRIU expects that "[vdso]" from /proc/../maps points at ELF/vdso image, rather than at VVAR data page. Laurent made a patch to keep CRIU working (by reading aux vector). But I think it still makes sence to separate two mappings into different VMAs. It will also make ppc64 less "special" for userspace and as a side-bonus will make VVAR page un-writable by debugger (which previously would COW page and can be unexpected). I opportunistically Cc stable on it: I understand that usually such stuff isn't a stable material, but that will allow us in CRIU have one workaround less that is needed just for one release (v5.11) on one platform (ppc64), which we otherwise have to maintain. I wouldn't go as far as to say that the commit 511157a is ABI regression as no other userspace got broken, but I'd really appreciate if it gets backported to v5.11 after v5.12 is released, so as not to complicate already non-simple CRIU-vdso code. Thanks! [1]: checkpoint-restore/criu#1417 Cc: stable@vger.kernel.org # v5.11 Signed-off-by: Dmitry Safonov <dima@arista.com> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Christophe Leroy <christophe.leroy@csgroup.eu> Reviewed-by: Vincenzo Frascino <vincenzo.frascino@arm.com> # vDSO parts. Acked-by: Andrei Vagin <avagin@gmail.com> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Link: https://lore.kernel.org/r/f401eb1ebc0bfc4d8f0e10dc8e525fd409eb68e2.1617209142.git.christophe.leroy@csgroup.eu Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

BugLink: https://bugs.launchpad.net/bugs/1928857 commit 1c4bce6 upstream. Since commit 511157a ("powerpc/vdso: Move vdso datapage up front") VVAR page is in front of the VDSO area. In result it breaks CRIU (Checkpoint Restore In Userspace) [1], where CRIU expects that "[vdso]" from /proc/../maps points at ELF/vdso image, rather than at VVAR data page. Laurent made a patch to keep CRIU working (by reading aux vector). But I think it still makes sence to separate two mappings into different VMAs. It will also make ppc64 less "special" for userspace and as a side-bonus will make VVAR page un-writable by debugger (which previously would COW page and can be unexpected). I opportunistically Cc stable on it: I understand that usually such stuff isn't a stable material, but that will allow us in CRIU have one workaround less that is needed just for one release (v5.11) on one platform (ppc64), which we otherwise have to maintain. I wouldn't go as far as to say that the commit 511157a is ABI regression as no other userspace got broken, but I'd really appreciate if it gets backported to v5.11 after v5.12 is released, so as not to complicate already non-simple CRIU-vdso code. Thanks! [1]: checkpoint-restore/criu#1417 Cc: stable@vger.kernel.org # v5.11 Signed-off-by: Dmitry Safonov <dima@arista.com> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Christophe Leroy <christophe.leroy@csgroup.eu> Reviewed-by: Vincenzo Frascino <vincenzo.frascino@arm.com> # vDSO parts. Acked-by: Andrei Vagin <avagin@gmail.com> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Link: https://lore.kernel.org/r/f401eb1ebc0bfc4d8f0e10dc8e525fd409eb68e2.1617209142.git.christophe.leroy@csgroup.eu Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kamal Mostafa <kamal@canonical.com> Signed-off-by: Stefan Bader <stefan.bader@canonical.com>

BugLink: https://bugs.launchpad.net/bugs/1928184 commit 1c4bce6 upstream. Since commit 511157a ("powerpc/vdso: Move vdso datapage up front") VVAR page is in front of the VDSO area. In result it breaks CRIU (Checkpoint Restore In Userspace) [1], where CRIU expects that "[vdso]" from /proc/../maps points at ELF/vdso image, rather than at VVAR data page. Laurent made a patch to keep CRIU working (by reading aux vector). But I think it still makes sence to separate two mappings into different VMAs. It will also make ppc64 less "special" for userspace and as a side-bonus will make VVAR page un-writable by debugger (which previously would COW page and can be unexpected). I opportunistically Cc stable on it: I understand that usually such stuff isn't a stable material, but that will allow us in CRIU have one workaround less that is needed just for one release (v5.11) on one platform (ppc64), which we otherwise have to maintain. I wouldn't go as far as to say that the commit 511157a is ABI regression as no other userspace got broken, but I'd really appreciate if it gets backported to v5.11 after v5.12 is released, so as not to complicate already non-simple CRIU-vdso code. Thanks! [1]: checkpoint-restore/criu#1417 Cc: stable@vger.kernel.org # v5.11 Signed-off-by: Dmitry Safonov <dima@arista.com> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Tested-by: Christophe Leroy <christophe.leroy@csgroup.eu> Reviewed-by: Vincenzo Frascino <vincenzo.frascino@arm.com> # vDSO parts. Acked-by: Andrei Vagin <avagin@gmail.com> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Link: https://lore.kernel.org/r/f401eb1ebc0bfc4d8f0e10dc8e525fd409eb68e2.1617209142.git.christophe.leroy@csgroup.eu Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Paolo Pisati <paolo.pisati@canonical.com>

adrianreber mentioned this issue Mar 24, 2021

test: fix for cgroup00 failures on cgroup2 systems #1415

Closed

github-actions bot added the stale-issue label Apr 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VDSO errors on ppc64le on 5.11.x #1417

VDSO errors on ppc64le on 5.11.x #1417

adrianreber commented Mar 24, 2021

rppt commented Mar 24, 2021

ldu4 commented Mar 25, 2021

ldu4 commented Mar 25, 2021 •

edited

Loading

chleroy commented Mar 25, 2021

ldu4 commented Mar 25, 2021

0x7f454c46 commented Mar 25, 2021 •

edited

Loading

0x7f454c46 commented Mar 25, 2021

ldu4 commented Mar 29, 2021

github-actions bot commented Apr 29, 2021

adrianreber commented May 12, 2021 •

edited

Loading

VDSO errors on ppc64le on 5.11.x #1417

VDSO errors on ppc64le on 5.11.x #1417

Comments

adrianreber commented Mar 24, 2021

rppt commented Mar 24, 2021

ldu4 commented Mar 25, 2021

ldu4 commented Mar 25, 2021 • edited Loading

chleroy commented Mar 25, 2021

ldu4 commented Mar 25, 2021

0x7f454c46 commented Mar 25, 2021 • edited Loading

0x7f454c46 commented Mar 25, 2021

ldu4 commented Mar 29, 2021

github-actions bot commented Apr 29, 2021

adrianreber commented May 12, 2021 • edited Loading

ldu4 commented Mar 25, 2021 •

edited

Loading

0x7f454c46 commented Mar 25, 2021 •

edited

Loading

adrianreber commented May 12, 2021 •

edited

Loading