SSL protocol issues when fetching the PR

[wilzbach]

fetching https://github.com/wilzbach/phobos.git enforce-2 (attempt: 1/3)
error: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version while accessing https://github.com/wilzbach/phobos.git/info/refs?service=git-upload-pack
fatal: HTTP request failed
fetching https://github.com/wilzbach/phobos.git enforce-2 (attempt: 2/3)
error: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version while accessing https://github.com/wilzbach/phobos.git/info/refs?service=git-upload-pack
fatal: HTTP request failed
fetching https://github.com/wilzbach/phobos.git enforce-2 (attempt: 3/3)
error: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version while accessing https://github.com/wilzbach/phobos.git/info/refs?service=git-upload-pack
fatal: HTTP request failed

https://auto-tester.puremagic.com/show-run.ghtml?projectid=1&runid=3025521&isPull=true

[braddr]

Yes, github had a problem earlier today. This showed up across the fleet. There's already re-tries, as you can see in the log snippit, but at some point it just has to give up and declare failure. github has already gone back to working and the tests are passing currently.

[wilzbach]

This is happening again:

fetching https://github.com/kinke/dmd.git cpp2079 (attempt: 1/3)
fatal: unable to access 'https://github.com/kinke/dmd.git/': error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
fetching https://github.com/kinke/dmd.git cpp2079 (attempt: 2/3)
fatal: unable to access 'https://github.com/kinke/dmd.git/': error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
fetching https://github.com/kinke/dmd.git cpp2079 (attempt: 3/3)
fatal: unable to access 'https://github.com/kinke/dmd.git/': error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version

dlang/dmd#7940

And the status page from GitHub shows

All systems reporting at 100%

https://status.github.com/messages

[kinke]

[I had troubles too when pushing to my GitHub repo earlier.]

[braddr]

Looks like github has disabled older tls protocols, requiring tlsv1.2. Unfortunately freebsd 8.x (and 9.x) don't have a new enough openssl to support newer tls protocols. So, we're roughly in a position where github is forcing our hands in terms of upgrading to newer versions of freebsd. A bigger unfortunately, no one has ever finished making dmd and related packages pass the builds and tests for freebsd 10 and 11. So, dlang is in a bad spot.

https://githubengineering.com/crypto-removal-notice/

I suggest dropping freebsd 8.x support from the master and stable branch testers. If when someone gets the freebsd 10/11 builds and tests passing, they can re-enter the master and stable branches.

Additionally, if someone is really hung-ho, figuring out how to update freebsd 8.x to a modern version of openssl without breaking the rest of the os,

Lastly, I don't see an escape hatch on github to allow continued use of older protocols, but maybe they left one and haven't advertised it.

[wilzbach]

I suggest dropping freebsd 8.x support from the master and stable branch testers. If when someone gets the freebsd 10/11 builds and tests passing, they can re-enter the master and stable branches.

I presume that's the best way to go.

@jmdavis or @dkgroot might know a bit more about the current status of DMD on a modern BSD* OS?

[braddr]

@MartinNowak @WalterBright @andralex

Heads up guys, this is unfortunate but I'm not shocked that it's come to a forcing function.

[braddr]

I think I've got freebsd disabled for master and stable but still enabled for the "DMD upd fbsd" project.

[WalterBright]

We just don't have the resources to support older FreeBSD versions, and them not being compatible with github is just the nail in the coffin.

I propose moving to the oldest FreeBSD that works with github and abandoning the older ones.

[braddr]

We're at the point where there is NO version of freebsd that is supported that DMD works on out of the box. So, today, I yanked it from the all build configurations for the auto-tester except the "DMD upd fbsd". That config is doing test builds with freebsd 10 and 11, but they don't pass.

Someone is going to have to finally dig into and fix the problems. Until then, we no longer support freebsd as a tested platform.

[jmdavis]

Right now, it just looks like only 32-bit is failing. If that's true, can we at least have a 64-bit autotester setup running FreeBSD 11.1 so that PRs don't get merge which break it? Unfortunately, 32-bit is then still screwed for the moment, but that seems better to me than dropping all FreeBSD auto-testing entirely.

Given that I use FreeBSD as my primary platform, I've definitely tried to make sure that the latest version of FreeBSD works with the druntime and Phobos tests (either by fixing problems are pushing others to), but I'm not a compiler dev, so I pretty much never run the dmd test suite, and I don't even notice when that breaks. I also don't run any 32-bit systems, so I don't catch any problems there. Clearly, we as a group need to do a better job making sure that the autotester is able to run the latest FreeBSD and is doing so, otherwise, we're going to be consistently screwed on this.

In any case, clearly, one or more of the compiler devs is going to need to fix dmd so that its test suite passes so that we can properly support 32-bit FreeBSD. But based on the upd fbsd section on the autotester, it looks like we can at least have a 64-bit autotester setup.

[WalterBright]

Thank you, @braddr
Looking at the failure to build on FreeBSD 32:

/usr/local/bin/ld: skipping incompatible //usr/lib/libpthread.so when searching for -lpthread
/usr/local/bin/ld: skipping incompatible //usr/lib/libpthread.a when searching for -lpthread
/usr/local/bin/ld: cannot find -lpthread
/usr/local/bin/ld: skipping incompatible //usr/lib/libm.so when searching for -lm
/usr/local/bin/ld: skipping incompatible //usr/lib/libm.a when searching for -lm
/usr/local/bin/ld: cannot find -lm
/usr/local/bin/ld: skipping incompatible /usr/local/lib/gcc6/gcc/x86_64-portbld-freebsd10.3/6.4.0/libgcc.a when searching for -lgcc
/usr/local/bin/ld: skipping incompatible //usr/lib/libgcc.a when searching for -lgcc
/usr/local/bin/ld: cannot find -lgcc
/usr/local/bin/ld: skipping incompatible /usr/local/lib/gcc6/gcc/x86_64-portbld-freebsd10.3/6.4.0/../../../libgcc_s.so when searching for -lgcc_s
/usr/local/bin/ld: skipping incompatible //usr/lib/libgcc_s.so when searching for -lgcc_s
/usr/local/bin/ld: cannot find -lgcc_s
/usr/local/bin/ld: skipping incompatible /lib/libc.so.7 when searching for /lib/libc.so.7
/usr/local/bin/ld: cannot find /lib/libc.so.7
/usr/local/bin/ld: skipping incompatible /usr/lib/libc_nonshared.a when searching for /usr/lib/libc_nonshared.a
/usr/local/bin/ld: cannot find /usr/lib/libc_nonshared.a
/usr/local/bin/ld: skipping incompatible /usr/lib/libssp_nonshared.a when searching for /usr/lib/libssp_nonshared.a
/usr/local/bin/ld: cannot find /usr/lib/libssp_nonshared.a

It looks like the correct system libraries are not installed on the autotester machine.

[WalterBright]

@jmdavis yes, the autotester is passing on FreeBSD 64, no reason to not keep that platform.

[braddr]

https://auto-tester.puremagic.com/platform-history.ghtml?projectid=15&os=FreeBSD_64_64

I has passed the most recent iteration, but fails more often than not. And it doesn't look to be 10.x or 11.x specific. Am I missing something?

[WalterBright]

The random dmd failures are:

/home/ec2-user/sandbox/at-client/master-139146-FreeBSD_64_64/dmd/generated/freebsd/release/64/dmd -conf= -m64 -Irunnable  -fPIC -L-lstdc++  -odgenerated/runnable -ofgenerated/runnable/externmangle_0  runnable/externmangle.d generated/runnable/externmangle.cpp.o 
generated/runnable/externmangle.cpp.o: In function `Test10Dtor(Test10*&)':
externmangle.cpp:(.text+0xd8): undefined reference to `operator delete(void*, unsigned long)'
generated/runnable/externmangle.cpp.o: In function `Expression::dispose(Expression*&)':
externmangle.cpp:(.text+0x332): undefined reference to `operator delete(void*, unsigned long)'
generated/runnable/externmangle.cpp.o: In function `Test38::dispose(Test38*&)':
externmangle.cpp:(.text+0x436): undefined reference to `operator delete(void*, unsigned long)'
cc: error: linker command failed with exit code 1 (use -v to see invocation)

Why that would occur randomly makes no sense to me.

[WalterBright]

The phobos tests fail with:

gmake[1]: *** [posix.mak:348: generated/freebsd/debug/64/unittest/std/traits.o] Killed

a fine, unhelpful message.

[WalterBright]

This looks relevant: https://issues.dlang.org/show_bug.cgi?id=17596

[jmdavis]

This looks relevant: https://issues.dlang.org/show_bug.cgi?id=17596

That will matter when 12.0 comes out but doesn't affect 11.1. It has to do with an API change to the system calls having to do with inodes, since they change them from 32-bit to 64-bit, and those changes weren't done in time for 11. It is on my todo list to try and fix that though, since at the moment, I can't update my TrueOS machines because of it.

[braddr]

The delete related linker problem in the dmd test seems to be a freebsd 10.3 problem. It always fails on 10.3 and never fails for 11.1 (for a sample size of about 25 builds).

[WalterBright]

Should we update the test box to the latest FreeBSD? I'm good with that.

[braddr]

Either I'm misunderstanding your question or you're misunderstanding the state of things.

The two build machines in the "DMD upd fbsd" project are running freebsd 10.3 and 11.1. The several machines that were in the other build fleets were 8.3 and 8.4. The latter became non-functional today. The former aren't able to reliably build and run the tests.

What change are you suggesting?

I can't advise that the 10.3 and 11.1 build hosts be added to the main build fleet until the "DMD upd fbsd" boxes reliably pass the build and test steps. Otherwise all pulls are going to fail and require either manual pulling due to not being able to pass the full set of platforms or fixing the freebsd problems first. The former is really not reasonable. The latter isn't better than fixing them before adding new builders to the master and stable builds.

Do you feel differently?

[WalterBright]

Do you feel differently?

I was responding to "a freebsd 10.3 problem. It always fails on 10.3 " and "That will matter when 12.0 comes out but doesn't affect 11.1. It has to do with an API change to the system calls having to do with inodes, since they change them from 32-bit to 64-bit, and those changes weren't done in time for 11."

To me that suggests: "frak the older FreeBSD versions, and install the latest on the autotester machines."

[braddr]

I'm fine with ignoring 10.3, though seems premature to write off an entire in-production version with as little investigation as has been done so far. For all we know at this point 10.x might be easier to fix support for than 11.x. 12.x isn't released yet, so I don't suggest that being our base platform -- also haven't ever tested it, so unknown status with respect to dmd. That leaves 11.1, which also isn't passing yet.

[WalterBright]

Do you have a suggestion for "It always fails on 10.3" with some sort of linker problem?

[braddr]

Everything I know came from a quick glance at the logs to corrolate failures between platforms. If you want my suggestion, drop freebsd. Not worth the time investment relative to the number of contributors.

Regardless, I'm ready to close this ticket now that the defunct freebsd 8.x platform is removed from the tester fleet. The issues are not related to the auto-tester and keeping it in it's issue tracker mis-places the nature of the problem and the responsibility for owning the resolution. Should 10.x and/or 11.x become functional enough to add into the official build fleet, creating an issue for that would be appropriate.

[jmdavis]

If we're getting random failures on 64-bit 11.1, then we can't enable that yet. If we're getting random failures with 64-bit 10.3, then that doesn't give me warm, fuzzy feelings, but Walter and Andrei have previously stated that we can only afford to support the latest FreeBSD, and that would be 11.1. And if we can have an auto-tester running that, then we'll catch problems that won't be caught otherwise, even if that unfortunately means that 10.3 might have problems. So, if 64-bit 11.1 is running well enough to update the auto-tester for that, then I say go for it, but I don't know how well it's running. The upd fbsd section on the auto-tester just gives the architectures, not the versions, so I don't know which of the boxes are running 10.3 and which are running 11.1.

If you want to close this issue on the grounds that it's for fixing the 8.x boxes, and that's not possible, then I don't care. But I really don't want to see FreeBSD dropped from the auto-tester permanently.

I'd suggest that you make both of the boxes on the upd fbsd section run FreeBSD 11.1 and that we just forget about 10.3. Then we can see how stable 11.1 is and figure out when it's stable enough to re-enable as part of the auto-tester proper.

[dkgroot]

It's not 100% clear which os-patch level you are running with freebsd 11.1 / 10.3.
Checkout:
https://www.cyberciti.biz/open-source/update-your-openssl-on-freebsd-10-x-11-x-to-fix-vulnerabilities/

It's a pretty simple procedure to regularly run

freebsd-update fetch
freebsd-update install

When this causes the patchlevel to go up, a reboot is required.

freebsd-version -k    # shows kernel patchlevel
freebsd-version -u    # shows userland patchlevel
uname -mrs            # shows kernel patchlevel

After a reboot do run:

pkg update && pkg upgrade

That should solve the openssl issue you are seeing, and fix future issues as well.

More about freebsd updating/upgrading and patchlevels

[jmdavis]

@dkgroot Brad knows full-well how to update FreeBSD. That's not the problem.

The auto-tester has been running FreeBSD 8.x for ages now, and the oldest version of FreeBSD that's supported by the FreeBSD folks is 10.3. Brad has a couple of boxes running 10.3 or 11.1 (it's not clear which box is running what, because they're just marked as 32-bit and 64-bit in the auto-tester interface) listed under the upd fbsd tab: https://auto-tester.puremagic.com/?projectid=15

The problem is that while 8.x has been tested by the auto-tester for ages, so it's reasonably stable, newer versions have generally only been tested by folks locally, so problems have crept in and haven't all been fixed. So, Brad's extra testers running with 10.3 and 11.1 show problems that need to be fixed before the main auto-tester boxes can be updated. But we can't continue to use 8.x, because it's not receiving updates and does not support the version of SSL/TLS that github now insists on.

However, because the main auto-testers have not been running a recent version of FreeBSD, and there hasn't been enough of an effort to make sure that the problems found by the boxes under the upd freebsd have been fixed, Brad can't update the main auto-tester boxes, because the dmd tests will fail.

So, it looks like we're forced to disable FreeBSD on the main auto-tester boxes until the problems found by the more up-to-date auto-tester boxes are fixed.

[dkgroot]

@jmdavis I cannot know what other people know or do not know :-) Regarding the SSL issue (subject), it was not clear which patchlevel was running on these test-slaves. Adding the extra background information about patchlevels could help others (non freebsd users) understand what this was all about.

Deprecating old/eol versions ought be part of the process. Even 10.3 and 10.4 have been declared legacy :-) No need to keep 8.x around.

If there is anything i could do to help, just let me know.

[MartinNowak]

There are folks who care about 32-bit FreeBSD. Personally, I don't expect to ever use it, but 32-bit FreeBSD is definitely a thing - especially when embedded systems come into play.

The question we should ask is not whether it's a thing, but whether we're interested in it and whether dmd has to serve this niche. As clang is the platforms default compiler toolchain, addressing the platform with ldc would prolly make more sense. Then again 32-bit codegen is already in dmd and differences between ELF based systems are rather small.

[MartinNowak]

Guess what dmd runs to do the link step by default? Right, gcc.

@braddr, I actually fixed that over a year ago, dlang/dmd#4843. What dmd version are you using?
Mainly motivated by the horrible default experience on FreeBSD (requiring you to install gcc or use the env var).

[WalterBright]

he's working on fixing it

I am at the moment. It's turning out to be more complex than it initially looks. My experiments with clang show that sometimes the size of an empty struct is 0, sometimes it is not. The clang user never sees this because it is all hidden behind __builtin compiler magic, which I will need to come up with an equivalent for.

[WalterBright]

I'm fine with temporarily disabling the 32-bit FreeBSD auto-tester

A much less drastic thing is to temporarily disable the failing test, which I will re-enable when I get a fix ready.

[WalterBright]

Temporary workaround dlang/dmd#7952

[braddr]

To answer the question of what version is the HOST_DC, two obvious answers:

1. check the build log, we added that information block for a reason.
2. every auto-tester boot straps with the same version, 2.068.2.

[braddr]

Hopefully the last issue: https://issues.dlang.org/show_bug.cgi?id=18519

[wilzbach]

What dmd version are you using?
every auto-tester boot straps with the same version, 2.068.2.

FWIW there's nothing stopping you from upgrading the host compiler:

https://forum.dlang.org/post/zbxaviowooiaenaawmgs@forum.dlang.org

(there's SemaphoreCI in place to check that DMD is still compilable with the latest GDC + LDC release)

Hopefully the last issue: https://issues.dlang.org/show_bug.cgi?id=18519

I never used FreeBSD, so I can't really help here. Sorry. That's why I suggested to drop 32-bit (or at least temporarily disabling it).

[wilzbach]

So can we do something about all PRs being red and not mergeable? The way I see it no one cares enough about FreeBSD 32-bit, so can we please disable it until someone who cares comes along? This undermines our entire CI process for which we worked hard to maintain some sense of reliability and leaves a very bad public image.

[WalterBright]

I'm working on the 32 bit struct issue, though a workaround has already been pulled. If there are more issues with FreeBSD 32, I suggest pulling workarounds (like disabling the test) instead of pulling the whole platform.

[wilzbach]

I suggest pulling workarounds (like disabling the test) instead of pulling the whole platform.

-> dlang/phobos#6232

[wilzbach]

@braddr @WalterBright the workaround at dmd was targeted at master, not stable, so stable is still broken. Maybe we can disable the FreeBSD 32-bit tester for stable (until the next branch-off) or is that not on the table either?

[wilzbach]

FYI after merging the 20th PR manually because apparently I'm the one of the few who can do an admin override, I remove the CI requirement check for auto-tester at the GitHub branch protection, i.e. it's not longer a requirement for a PR.
The auto-merge feature won't merge a PR if it has a failing check.

---

Though I can only repeat that if the CI fails for all PRs this has severe downsides and makes people loose the tiny trust we slowly managed to build up over the last months, leaves newcomers and irregular contributors puzzled, etc.

[WalterBright]

Now the autotester appears to not run FreeBSD at all, so I can't test my PR to fix it.

[braddr]

It looks like master freebsd/32 builds are passing, which is good progress. That stable builds are failing indicates that there's one or more pulls that are only in master that need to be replicated into stable.

Walter, not sure what you're seeing, the builders are running jobs just like they're supposed to. I need to convert a couple of the old hosts over to 11.1 hosts so that the fleet is up to full strength. That's a throughput issue, not a not-running issue. I've held off in case we needed to revert, but I think we're past that now.

I agree that the path chosen to accomplish the 8.x -> 11.x migration of freebsd wasn't what I'd have done if I was in charge, but I also can't argue with the likelihood that if it'd been left disabled until done or some other less in-your-face approach that it might well have never gotten done.

[wilzbach]

It looks like master freebsd/32 builds are passing, which is good progress. That stable builds are failing indicates that there's one or more pulls that are only in master that need to be replicated into stable.

dlang/dmd#7970
dlang/phobos#6237

I agree that the path chosen to accomplish the 8.x -> 11.x migration of freebsd wasn't what I'd have done if I was in charge

Well, what's done is done, but maybe we can consider a path that doesn't involve breaking all PRs for a couple of days and me force-merging > 20 PRs?
For example, there could have been a special freebsd stage for this or a check for a label.
Even checking the title for "[FreeBSD]" and only then run the test on the FreeBSD hosts would of worked... maybe next time ;-)

but I also can't argue with the likelihood that if it'd been left disabled until done or some other less in-your-face approach that it might well have never gotten done.

Yes. Investing work for legacy systems isn't fun - especially if you don't use them.