build randomly fails with automake-1.16 #529

bmwiedemann · 2019-02-28T12:10:48Z

When building heimdal-7.5.0 in openSUSE with make -j1, we get

Making all in hcrypto
make[2]: Entering directory '/home/abuild/rpmbuild/BUILD/heimdal-7.5.0/lib/hcrypto'
  CC       test_rand.o
In file included from test_rand.c:42:
rand.h:46:10: fatal error: hcrypto/engine.h: No such file or directory
 #include <hcrypto/engine.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [Makefile:2064: test_rand.o] Error 1
make[2]: Leaving directory '/home/abuild/rpmbuild/BUILD/heimdal-7.5.0/lib/hcrypto'
make[1]: *** [Makefile:564: all-recursive] Error 1

to reproduce:

osc checkout openSUSE:Factory/libheimdal && cd $_
osc build --noservice -j1

I think there is some dependency not specified in lib/hcrypto/Makefile.am and when doing a parallel build, it usually gets created in time from a parallel job.

This bug was found while working on reproducible builds for openSUSE.

The text was updated successfully, but these errors were encountered:

MilhouseVH · 2019-06-18T04:55:48Z

I see this same failure on Ubuntu 17.10 when building heimdal-7.7.0 with automake-1.16 and -j8.

I have no problem building heimdal-7.7.0 with automake-1.15.1 (with -j8), so there's an issue when building with automake-1.16 (also automake-1.16.1).

@bmwiedemann what version of automake are you using?

bmwiedemann · 2019-06-18T08:20:05Z

We are building with automake-1.16.1 in openSUSE:Factory. Building for Leap:15.1 with automake-1.15.1 indeed does not fail in this way. So maybe libheimdal triggers a problem/regression in automake-1.16 ?

MilhouseVH · 2019-06-18T19:52:55Z

The following automake commit seems to be the trigger for the heimdal failure:

http://git.savannah.gnu.org/cgit/automake.git/commit/?id=f4e91bfc490da63209aad19636568da3b955dcd4&h=master

I can apply this commit on automake-1.15.1 and reproduce the above failure.

I've no idea if this is a heimdal or automake issue.

MilhouseVH · 2019-07-06T02:34:40Z

@bmwiedemann it might be worth updating the title of this issue to reference automake-1.16+ as I don't think the problem has anything to do with -j1.

MilhouseVH · 2019-10-06T01:15:23Z

@nicowilliams sorry for the ping, but any ideas on this one? automake-1.16 is starting to be used more widely (it's been available for 19 months) and heimdal isn't compatible - the lack of any similar reports against automake would suggest this is a heimdal issue...

jaltman · 2019-10-06T03:39:27Z

@MilhouseVH my opinion is that since your have identified the specific change which introduced a regression in automake 1.16 that the bug report be filed against automake. The commit message for http://git.savannah.gnu.org/cgit/automake.git/commit/?id=f4e91bfc490da63209aad19636568da3b955dcd4&h=master indicates that there should be no change in behavior. Shorter pathnames are supposed to be used when doing so would not introduce an ambiguity. Clearly that change breaks Heimdal. Notifying the author of the unintended side effect of the commit is the appropriate thing to do.

Unfortunately Heimdal will need to work around this regression as well.

MilhouseVH · 2019-10-06T16:29:06Z

@jaltman thanks for the reply. And thanks for prompting me to dig a little deeper - I have in fact now found an automake email discussion[1] where the subdir-objects commit[2] has been acknowledged as causing a regression[3], although not in exactly the same way that it fails for heimdal but maybe close enough (I myself don't have an issue building gnulib with automake-1.16.1 and GNU make-4.2.1).

There may be a work-around[4] but I'm not entirely sure how this can be applied to heimdal as to be quite honest I have limited understanding of Makefile syntax - right now, I'm just stuck in the middle... 😄

I've sent a reply to the automake mailing list (a reply to Thomas Martitz[3] - I've subscribed, but no idea when my post will appear) with details of this heimdal issue in the hope of a potential workaround, but also to inform any potential fix.

MilhouseVH · 2019-10-06T17:06:37Z

My mailing list post is here:

https://lists.gnu.org/archive/html/automake/2019-10/msg00000.html

MilhouseVH · 2020-03-22T02:34:42Z

This heimdal build failure continues to be an issue with the newly released automake-1.16.2.

Either this isn't an automake issue, or automake don't consider it an issue worth fixing - could the heimdal developers possibly reach out to automake directly to try and resolve this?

lhoward · 2020-04-07T04:58:55Z

The comments in the mailing list messages you referenced suggest it's something to do with pathname equivalences, e.g. ./foo/libbar.a not matching foo/libbar.a.

lhoward · 2020-04-07T04:59:22Z

I'm trying with automake-1.16.2 on macOS but haven't been able to duplicate the issue yet.

lhoward · 2020-04-07T05:45:40Z

With automake-1.16.2 (and latest autoconf/libtool) on Ubuntu 18.04.3 LTS and Heimdal master I can't seem to duplicate this using make -j1. I wonder if the issue is that in your build system the built artefacts go outside the source tree?

lhoward · 2020-04-07T06:06:56Z

OK. Could you please try grabbing commit cc6a3f3 and seeing if that fixes it?

bmwiedemann · 2020-04-07T08:03:26Z

Commit cc6a3f3 fixed it for me (still with automake-1.16.1). If others don't object, this issue can be closed.

lhoward · 2020-04-07T08:04:26Z

I'll leave it to @jaltman to close, in case he wants to cut another release.

MilhouseVH · 2020-04-07T12:46:44Z

@lhoward many thanks for taking a look at this!

I'm testing with automake-1.16.2 (also tested with automake-1.16.1 but result is the same) and unfortunately cc6a3f3 doesn't appear to be a complete fix, although it does move the failure on a step.

Previously, heimdal-7.7.0 (unpatched) with automake-1.16.2 would fail as follows:

Making all in hcrypto
make[3]: Entering directory '/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/build/heimdal-7.7.0/.x86_64-linux-gnu/lib/hcrypto'
  CC       test_rand.o
In file included from /home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/build/heimdal-7.7.0/lib/hcrypto/test_rand.c:42:0:
/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/build/heimdal-7.7.0/lib/hcrypto/rand.h:46:10: fatal error: hcrypto/engine.h: No such file or directory
 #include <hcrypto/engine.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [Makefile:2061: test_rand.o] Error 1
make[3]: Leaving directory '/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/build/heimdal-7.7.0/.x86_64-linux-gnu/lib/hcrypto'
make[2]: *** [Makefile:566: all-recursive] Error 1
make[2]: Leaving directory '/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/build/heimdal-7.7.0/.x86_64-linux-gnu/lib'
make[1]: *** [Makefile:614: all-recursive] Error 1
make[1]: Leaving directory '/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/build/heimdal-7.7.0/.x86_64-linux-gnu'

(build log: http://ix.io/2h34)

With cc6a3f3, heimdal-7.7.0 and automake-1.16.2 now fail as follows - heimdal seems to get a little further in the build than hcrypto:

make  all-am
make[4]: Entering directory '/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/build/heimdal-7.7.0/.x86_64-linux-gnu/lib/hx509'
  GEN    /home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/build/heimdal-7.7.0/lib/hx509/hx509-protos.h
updating hx509-protos.h
  GEN    /home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/build/heimdal-7.7.0/lib/hx509/hx509-private.h
updating hx509-private.h
  GEN    hxtool-commands.h
;  CC       hxtool.o
In file included from /home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/build/heimdal-7.7.0/lib/hx509/hxtool.c:34:0:
/home/ubuntu/projects/LibreELEC.tv/build.LibreELEC-RPi2.arm-9.80-devel/build/heimdal-7.7.0/lib/hx509/hx_locl.h:66:10: fatal error: ocsp_asn1.h: No such file or directory
 #include <ocsp_asn1.h>
          ^~~~~~~~~~~~~
compilation terminated.
make[4]: *** [Makefile:1320: hxtool.o] Error 1

(build log: http://ix.io/2h33, and build log with V=1: http://ix.io/2h3o)

ocsp_asn1.h has not been generated, and does not exist.

In both cases I'm building heimdal with export MAKEFLAGS=-j1

MilhouseVH · 2020-04-07T13:14:33Z

Testing automake-1.16.2 with/without cc6a3f3 and MAKEFLAGS=-j8 is actually successful!

-j8 with cc6a3f3: log
-j8 without cc6a3f3: log

Although I doubt it will always build reliably with -j8, as there is clearly race-related behaviour.

But with -j1, it always fails as per the previous post.

jaltman · 2020-04-07T13:49:17Z

Since "master" 7055365 doesn't experience the problem there must be more commits on master that fix build dependency issues that must be cherry-picked to "heimdal-7-1-branch".

jaltman · 2020-04-07T18:12:37Z

@MilhouseVH please try building c4cff68 which is the new tip of the heimdal-7-1-branch

lhoward · 2020-04-07T22:49:43Z

BTW OT but @bmwiedemann – when did SuSE switch back to Heimdal?

MilhouseVH · 2020-04-08T01:27:54Z

@jaltman I will build latest heimdal master and get back to you with feedback. Unfortunately my remote build server VM fell off the network a few hours ago which may delay me!

lhoward · 2020-04-08T01:29:57Z

@MilhouseVH I think @jaltman is asking you to build c4cff68 which is the new tip of the heimdal-7-1-branch, rather than master (which is not in a shippable state).

MilhouseVH · 2020-04-08T01:36:21Z

@jaltman ah OK thanks - as soon as I re-establish contact with my VM I'll give it a go!

MilhouseVH · 2020-04-08T20:26:49Z

@jaltman, @lhoward: I've successfully built c4cff68 (heimdal-7-1-branch) and can confirm this branch has no issue with automake-1.16.2 (and make-4.3, just for completeness). I tested both -j1 and also -j8, building both multiple times and had no failures. Many thanks!

jaltman · 2020-04-08T21:24:26Z

@MilhouseVH thanks for testing. I'm closing this issue.

bmwiedemann changed the title ~~build fails with make -j1~~ build randomly fails with automake-1.16 Jul 8, 2019

jaltman added the bug label Apr 7, 2020

jaltman added this to To do in Heimdal 8 Release via automation Apr 7, 2020

jaltman added this to the Heimdal 8 milestone Apr 7, 2020

lhoward self-assigned this Apr 7, 2020

jaltman linked a pull request Apr 7, 2020 that will close this issue

pull up automake fixes from master so proper dependencies exist even with newer automake #686

Merged

jaltman closed this as completed Apr 8, 2020

Heimdal 8 Release automation moved this from To do to Done Apr 8, 2020

MilhouseVH mentioned this issue Apr 22, 2020

ETA for new 7.7.x release (if possible)? #698

Open

heitbaum mentioned this issue Dec 28, 2020

packages/devel - next batch of bumps LibreELEC/LibreELEC.tv#4806

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build randomly fails with automake-1.16 #529

build randomly fails with automake-1.16 #529

bmwiedemann commented Feb 28, 2019

MilhouseVH commented Jun 18, 2019

bmwiedemann commented Jun 18, 2019

MilhouseVH commented Jun 18, 2019 •

edited

MilhouseVH commented Jul 6, 2019

MilhouseVH commented Oct 6, 2019

jaltman commented Oct 6, 2019 •

edited

MilhouseVH commented Oct 6, 2019 •

edited

MilhouseVH commented Oct 6, 2019

MilhouseVH commented Mar 22, 2020

lhoward commented Apr 7, 2020

lhoward commented Apr 7, 2020

lhoward commented Apr 7, 2020

lhoward commented Apr 7, 2020

bmwiedemann commented Apr 7, 2020

lhoward commented Apr 7, 2020

MilhouseVH commented Apr 7, 2020

MilhouseVH commented Apr 7, 2020

jaltman commented Apr 7, 2020

jaltman commented Apr 7, 2020

lhoward commented Apr 7, 2020

MilhouseVH commented Apr 8, 2020

lhoward commented Apr 8, 2020

MilhouseVH commented Apr 8, 2020

MilhouseVH commented Apr 8, 2020

jaltman commented Apr 8, 2020

build randomly fails with automake-1.16 #529

build randomly fails with automake-1.16 #529

Comments

bmwiedemann commented Feb 28, 2019

MilhouseVH commented Jun 18, 2019

bmwiedemann commented Jun 18, 2019

MilhouseVH commented Jun 18, 2019 • edited

MilhouseVH commented Jul 6, 2019

MilhouseVH commented Oct 6, 2019

jaltman commented Oct 6, 2019 • edited

MilhouseVH commented Oct 6, 2019 • edited

MilhouseVH commented Oct 6, 2019

MilhouseVH commented Mar 22, 2020

lhoward commented Apr 7, 2020

lhoward commented Apr 7, 2020

lhoward commented Apr 7, 2020

lhoward commented Apr 7, 2020

bmwiedemann commented Apr 7, 2020

lhoward commented Apr 7, 2020

MilhouseVH commented Apr 7, 2020

MilhouseVH commented Apr 7, 2020

jaltman commented Apr 7, 2020

jaltman commented Apr 7, 2020

lhoward commented Apr 7, 2020

MilhouseVH commented Apr 8, 2020

lhoward commented Apr 8, 2020

MilhouseVH commented Apr 8, 2020

MilhouseVH commented Apr 8, 2020

jaltman commented Apr 8, 2020

MilhouseVH commented Jun 18, 2019 •

edited

jaltman commented Oct 6, 2019 •

edited

MilhouseVH commented Oct 6, 2019 •

edited