Update CI from ubuntu 16.04 (EOL) to 19.04

[gnzlbg]

Closes #1333 .

[rust-highfive]

@gnzlbg: no appropriate reviewer found, use r? to override

[gnzlbg]

Let's see if including the bits/syscall.h header fixes those!

[gnzlbg]

So that did not work, that header cannot be included directly.

[mati865]

It seems SYS_syscalls is no longer generated, at least it's absent in 18.04 container with libc6-dev and linux-headers-5.0.0 installed.

[gnzlbg]

That would happen if the kernel does not define __NR_syscalls, but AFAICT it does ? Maybe syscalls is just missing from the list that glibc uses.

[gnzlbg]

Yes, that appears to be the case: https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=2dba5ce7b8115d6a2789bf279892263621088e74;hp=5f3b183d198b39ca993a41aadb02bddd9fde078d

Shows that "pseudo-system calls" that define ranges, and unused / reserved system calls, are skipped, which were not being skipped before. So code that uses them now fails to compile with newer glibc versions.

[gnzlbg]

AFAICT, that's a backwards incompatible breaking change from glibc, although maybe the behavior was broken before already.

[gnzlbg]

cc @fweimer

[mati865]

Seems it's not present for some architectures:

rg "__NR_syscalls" /usr/
/usr/include/asm-generic/unistd.h
736:#undef __NR_syscalls
737:#define __NR_syscalls 292
781:#undef __NR_syscalls
782:#define __NR_syscalls (__NR3264_lstat+1)
799:#undef __NR_syscalls
800:#define __NR_syscalls (__NR_signalfd+1)
834:#undef __NR_syscalls
835:#define __NR_syscalls (__NR_mmap+1)
897:#undef __NR_syscalls
898:#define __NR_syscalls (__NR_fork+1)

/usr/src/linux-headers-5.0.0-15/arch/arm/tools/syscallnr.sh
29:        echo "#define __NR_syscalls $nr"

/usr/src/linux-headers-5.0.0-15/include/uapi/asm-generic/unistd.h
744:#undef __NR_syscalls
745:#define __NR_syscalls 295

/usr/src/linux-headers-5.0.0-15/arch/alpha/kernel/syscalls/syscallhdr.sh
32:     printf "#define __NR_syscalls\t%s\n" "${nxt}"

/usr/src/linux-headers-5.0.0-15/arch/alpha/include/asm/unistd.h
7:#define NR_SYSCALLS   __NR_syscalls

/usr/src/linux-headers-5.0.0-15/arch/arm64/include/asm/unistd.h
56:#define NR_syscalls (__NR_syscalls)

/usr/src/linux-headers-5.0.0-15/arch/ia64/kernel/syscalls/syscallhdr.sh
32:     printf "#define __NR_syscalls\t%s\n" "${nxt}"

/usr/src/linux-headers-5.0.0-15/arch/microblaze/kernel/syscalls/syscallhdr.sh
32:     printf "#define __NR_syscalls\t%s\n" "${nxt}"

/usr/src/linux-headers-5.0.0-15/arch/m68k/kernel/syscalls/syscallhdr.sh
32:     printf "#define __NR_syscalls\t%s\n" "${nxt}"

/usr/src/linux-headers-5.0.0-15/arch/parisc/kernel/syscalls/syscallhdr.sh
32:     printf "#define __NR_syscalls\t%s\n" "${nxt}"

/usr/src/linux-headers-5.0.0-15/arch/ia64/include/asm/unistd.h
13:#define NR_syscalls          __NR_syscalls /* length of syscall table */

/usr/src/linux-headers-5.0.0-15/arch/riscv/include/asm/unistd.h
23:#define NR_syscalls (__NR_syscalls)

/usr/src/linux-headers-5.0.0-15/arch/parisc/include/asm/unistd.h
7:#define __NR_Linux_syscalls   __NR_syscalls

/usr/src/linux-headers-5.0.0-15/arch/m68k/include/asm/unistd.h
7:#define NR_syscalls   __NR_syscalls

/usr/src/linux-headers-5.0.0-15/arch/powerpc/kernel/syscalls/syscallhdr.sh
32:     printf "#define __NR_syscalls\t%s\n" "${nxt}"

/usr/src/linux-headers-5.0.0-15/arch/arm/include/asm/syscall.h
18:#define NR_syscalls (__NR_syscalls)

/usr/src/linux-headers-5.0.0-15/arch/mips/kernel/syscalls/syscallhdr.sh
32:     printf "#define __NR_syscalls\t%s\n" "${nxt}"

/usr/src/linux-headers-5.0.0-15/arch/sh/kernel/syscalls/syscallhdr.sh
32:     printf "#define __NR_syscalls\t%s\n" "${nxt}"

/usr/src/linux-headers-5.0.0-15/arch/powerpc/include/asm/unistd.h
14:#define NR_syscalls  __NR_syscalls

/usr/src/linux-headers-5.0.0-15/arch/sparc/kernel/syscalls/syscallhdr.sh
32:     printf "#define __NR_syscalls\t%s\n" "${nxt}"

/usr/src/linux-headers-5.0.0-15/arch/xtensa/kernel/syscalls/syscallhdr.sh
32:     printf "#define __NR_syscalls\t%s\n" "${nxt}"

/usr/src/linux-headers-5.0.0-15/arch/sh/include/asm/unistd.h
8:#define NR_syscalls   __NR_syscalls

/usr/src/linux-headers-5.0.0-15/arch/xtensa/include/asm/unistd.h
25:#define NR_syscalls                          __NR_syscalls

/usr/src/linux-headers-5.0.0-15/arch/sparc/include/asm/unistd.h
20:#define NR_syscalls  __NR_syscalls

/usr/src/linux-headers-5.0.0-15/arch/arc/include/uapi/asm/unistd.h
30:#define NR_syscalls  __NR_syscalls

/usr/src/linux-headers-5.0.0-15/arch/sh/include/uapi/asm/unistd_64.h
420:#define __NR_syscalls               394

/usr/src/linux-headers-5.0.0-15/arch/sh/include/uapi/asm/unistd_32.h
400:#define __NR_syscalls               383

[gnzlbg]

@mati865 the error is (e.g. on aarch64):

error: 'SYS_arch_specific_syscall' undeclared here (not in a function); did you mean '__NR_arch_specific_syscall'?
cargo:warning=             static const  long __test_const_SYS_arch_specific_syscall_val = SYS_arch_specific_syscall;
cargo:warning=                                                                             ^~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=                                                                             __NR_arch_specific_syscall
cargo:warning=/checkout/target/aarch64-unknown-linux-gnu/debug/build/libc-test-bf57c51a205470a0/out/main.c:29556:64: error: 'SYS_syscalls' undeclared here (not in a function); did you mean '__NR_syscalls'?
cargo:warning=             static const  long __test_const_SYS_syscalls_val = SYS_syscalls;
cargo:warning=                                                                ^~~~~~~~~~~~
cargo:warning=                                                                __NR_syscalls

This means that SYS_syscalls is not defined in the headers, but __NR_syscalls is. Before glibc used to export SYS_syscalls for these targets. Now it does not, even though __NR_syscalls still exists.

[gnzlbg]

The same happens in other targets with the reserved/unused syscalls.

[gnzlbg]

So most build jobs are green now. I have no idea why s390x is failing, the error message is weird. For example, see:

cargo:warning=/checkout/target/s390x-unknown-linux-gnu/debug/build/libc-test-bf57c51a205470a0/out/main.c:24441:73: error: 'PTRACE_GETREGS' undeclared here (not in a function); did you mean 'PTRACE_GETREGSET'?
cargo:warning=             static const unsigned int __test_const_PTRACE_GETREGS_val = PTRACE_GETREGS;
cargo:warning=                                                                         ^~~~~~~~~~~~~~
cargo:warning=                                                                         PTRACE_GETREGSET
cargo:warning=/checkout/target/s390x-unknown-linux-gnu/debug/build/libc-test-bf57c51a205470a0/out/main.c:24447:73: error: 'PTRACE_SETREGS' undeclared here (not in a function); did you mean 'PTRACE_GETREGS'?
cargo:warning=             static const unsigned int __test_const_PTRACE_SETREGS_val = PTRACE_SETREGS;
cargo:warning=                                                                         ^~~~~~~~~~~~~~
cargo:warning=                                                                         PTRACE_GETREGS

Notice how the first error says that PTRACE_GETREGS is undeclared, but the suggestion of the second error does suggest using PTRACE_GETREGS, which can only happen if that has been declared somewhere =/

[mati865]

I was also puzzled by this one. Definitions seems to be absent though:

• PPC: https://elixir.bootlin.com/linux/latest/source/arch/powerpc/include/uapi/asm/ptrace.h#L182
• S390X: https://elixir.bootlin.com/linux/latest/source/arch/s390/include/uapi/asm/ptrace.h
• architectures with the define: https://elixir.bootlin.com/linux/latest/ident/PTRACE_GETREGS

[gnzlbg]

Most build jobs fail again with ubuntu 19.04, the errors are of the form:

bad ucontext_t size: rust: 936 (0x3a8) != c 968 (0x3c8)
bad fscanf function pointer: rust: 140035897326624 (0x7f5ca5ea7c20) != c 140035897330928 (0x7f5ca5ea8cf0)
bad scanf function pointer: rust: 140035897326816 (0x7f5ca5ea7ce0) != c 140035897330688 (0x7f5ca5ea8c00)
bad sscanf function pointer: rust: 140035897327024 (0x7f5ca5ea7db0) != c 140035897331136 (0x7f5ca5ea8dc0)
bad NFT_MSG_MAX value at byte 0: rust: 22 (0x16) != c 25 (0x19)

The incorrect const and field size are easily fixable, but the incorrect function pointers means that Rust points to a different function than C in our tests. Maybe a different version of glibc is being pulled via the rust standard library than the one C is using ? No idea yet.

Also, the s390x build job fails with:

/prog: /lib/libc.so.6: version `GLIBC_2.27' not found (required by /prog)

but AFAICT ubuntu 19.04 ships glibc 2.29 =/

[gnzlbg]

So everything is green now, except for the wasi build jobs which still fail with target not found (cc @alexcrichton @sunfishcode) and the android build jobs which are allowed to fail.

[gnzlbg]

So I've pulled the renaming changes only from #1342 .

[gnzlbg]

Now the documentation build job fails because bitrig was remove in the latest nightly, which was just released. Perfect timing.

[gnzlbg]

@bors: r+

[bors]

📌 Commit 8eceb62 has been approved by gnzlbg

[bors]

⌛ Testing commit 8eceb62 with merge 4ac8346...

[gnzlbg]

This was, by far, the largest game of code golf I've ever played.

[bors]

☀️ Test successful - checks-cirrus, checks-travis, status-appveyor
Approved by: gnzlbg
Pushing 4ac8346 to master...

[gnzlbg]

Thank you @mati865 for the feedback and help with the changes!

[alexcrichton]

@gnzlbg FWIW if this happens in the future I'd recommend simply disabling targets on CI. I think it's best to keep the wheels turning and the problems fixed here in general should be inconsequential because the current state of the crate isn't exactly full of bugs to the point that it's unusable. By enabling PRs to land quicker it can help reduce some stress from landing big changes/refactorings like this all at the same time.

[gnzlbg]

@alexcrichton The Ubuntu version that was being used went end-of-life and the Docker images for it were removed, so all build jobs using it (almost all of them) started failing. That is, we would have had to turn off CI completely. I suppose we could have done that, to be able to continue merging PRs without any testing on CI, but I don't think we should have done a new release until CI was up again, and at that point, merged PRs might have introduced errors, that I would have had to fix in the PR updating CI, so... I don't know if that would have been much better.

[alexcrichton]

@gnzlbg oh sure yeah I understand but it probably wasn't too hard to get the major targets working again on CI and the main difficulty was getting everything working, right? I was mostly just saying that this could probably get staged where the main targets are brought back up and then the remaining targets would be follow-ups.

[gnzlbg]

Well, maybe? The part that took most of the time here was getting x86_64 and aarch64 gnu Linux targets working (the Ubuntu upgrade updates glibc version, so mostly only the gnu targets were affected by it). This PR did not fix Android (there is a PR open for this) nor Emscripten, which are still broken, but getting the tests to pass on the Linux gnu targets without breaking the other targets more required splitting the Linux bits from the libc-test/build.rs, which was done as simple as possible (there are a million fixmes, which are now being cleaned up in other PRs). The other Linux targets (e.g. the musl ones) kind of just worked, or were easy to fix.

The sparc64 and s390x fixes were probably unnecessary and could have been done later, but they weren't very time consuming either.

What costed most of the time is having to test this on travis-ci, which gives me a 30min iteration time, because the docker containers that we have do not work on MacOS X.

[alexcrichton]

Ah ok, well whatever works works! Not being able to test locally I can also see would significantly complicate things...