Need to run ./autogen.sh twice

[dionyziz]

The first invocation of ./autogen.sh fails, causing ./configure and make to fail. I have to run ./autogen.sh twice. In that case, it works:

root@484cf9b0d581:/home/grr/grr/protobuf# ./autogen.sh
Google Test not present.  Fetching gtest-1.7.0 from the web...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1136k  100 1136k    0     0  10.0M      0 --:--:-- --:--:-- --:--:-- 11.8M
+ sed -i -e s/RuntimeLibrary="5"/RuntimeLibrary="3"/g;
           s/RuntimeLibrary="4"/RuntimeLibrary="2"/g; gtest/msvc/gtest-md.vcproj gtest/msvc/gtest.vcproj gtest/msvc/gtest_main-md.vcproj gtest/msvc/gtest_main.vcproj gtest/msvc/gtest_prod_test-md.vcproj gtest/msvc/gtest_prod_test.vcproj gtest/msvc/gtest_unittest-md.vcproj gtest/msvc/gtest_unittest.vcproj
+ autoreconf -f -i -Wall,no-obsolete
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `build-aux'.
libtoolize: copying file `build-aux/config.guess'
libtoolize: copying file `build-aux/config.sub'
libtoolize: copying file `build-aux/install-sh'
libtoolize: copying file `build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
configure.ac:57: installing `./ar-lib'
configure.ac:38: installing `./config.guess'
configure.ac:38: installing `./config.sub'
configure.ac:40: installing `./install-sh'
configure.ac:84: required file `./ltmain.sh' not found
configure.ac:40: installing `./missing'
src/Makefile.am: installing `./depcomp'
autoreconf: automake failed with exit status: 1
root@484cf9b0d581:/home/grr/grr/protobuf# ./autogen.sh
+ sed -i -e s/RuntimeLibrary="5"/RuntimeLibrary="3"/g;
           s/RuntimeLibrary="4"/RuntimeLibrary="2"/g; gtest/msvc/gtest-md.vcproj gtest/msvc/gtest.vcproj gtest/msvc/gtest_main-md.vcproj gtest/msvc/gtest_main.vcproj gtest/msvc/gtest_prod_test-md.vcproj gtest/msvc/gtest_prod_test.vcproj gtest/msvc/gtest_unittest-md.vcproj gtest/msvc/gtest_unittest.vcproj
+ autoreconf -f -i -Wall,no-obsolete
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `build-aux'.
libtoolize: copying file `build-aux/config.guess'
libtoolize: copying file `build-aux/config.sub'
libtoolize: copying file `build-aux/install-sh'
libtoolize: copying file `build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
libtoolize: putting auxiliary files in `.'.
libtoolize: copying file `./config.guess'
libtoolize: copying file `./config.sub'
libtoolize: copying file `./install-sh'
libtoolize: copying file `./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
+ rm -rf autom4te.cache config.h.in~
+ exit 0

This occurs in Ubuntu 12.04.

[dionyziz]

Note that this happens on a clean system only, and can be reproduced using Docker or Travis. If ./autogen.sh is ran after a make clean, it succeeds the first time.

[xfxyjwf]

I can not reproduce this problem on my machine and our Travis project is also passing: https://travis-ci.org/google/protobuf

Are you using the github master branch or some other version?

[dionyziz]

I'll see if I can make a minimal testcase.

[dionyziz]

I have a minimal testcase in the autogen_twice branch of my fork. I made two annotated tags on that repository:

• single, which contains a build file that runs ./autogen.sh once
• double, which contains a build file that runs ./autogen.sh once

The travis results can be seen in the following links.

Running ./autogen.sh once

When running ./autogen.sh once, the build fails as seen here:
https://travis-ci.org/dionyziz/protobuf/builds/46740409#L270

The error is:

make: *** No targets specified and no makefile found. Stop.

Running ./autogen.sh twice

When running ./autogen.sh twice, the build succeeds as seen here:
https://travis-ci.org/dionyziz/protobuf/builds/46740413#L892

Here is my modified install.sh script which also contains some diagnostic messages. It's just 5 lines, so I consider it minimal.

Here is my modified travis.yml file which simply calls "install.sh".

[xfxyjwf]

I have found a similar issue report here:
akheron/jansson#182

I tried adding "AC_CONFIG_AUX_DIR([.])" and it seems to fix the problem. Could you check whether it works for you as well?

[dionyziz]

@xfxyjwf Nope, still failing: https://travis-ci.org/dionyziz/protobuf/builds/47279041

Here's my diff: dionyziz@30e7b27

So I think this is a bug. Interestingly, it's hard to reproduce because you need to run ./autogen.sh on the system twice, but if you run make clean, it "remembers" that you ran autogen previously, so it works with only one run. So you need a really clean system to reproduce the problem.

This seems like two separate problems to me:

1. autogen.sh runs some things out-of-order and so is required to be ran twice under certain circumstances.
2. make clean doesn't fully clean everything that autogen.sh left behind, as after you've run autogen.sh followed by a make clean, you no longer need to run it twice.

Interestingly, it may help to note that I cannot reproduce the problem if I don't git clone and use the direct tree provided by travis (as you do in your default .travis.yml). This may point to the right direction for debugging; I suspect it may be related to the particular source tree ./autogen.sh is ran on. However, I cannot pinpoint the problem.

This bug is not blocking me, so no rush. I just have two ./autogen.sh in my build script and it works fine. It just seems like a bug and would be nice to see it fixed, especially because it's so esoteric.

[chrislgarry]

This is still reproducible. Clean install of Ubuntu 16.04 image on VirtualBox. Protobuf is the very first thing I try to install on this VM, aside from its dependencies and git/vim. Same exact error. ./autogen.sh || autogen.sh is the current work around.

[xfxyjwf]

@chrislgarry We have never been able to reproduce this issue in our testing environment. It would be great if you can provide a docker image that's able to reproduce the issue.

[chrislgarry]

Ok sure. I’ll try to do to that this weekend. 2018年7月21日(土) 2:09 Feng Xiao <notifications@github.com>:

…

@chrislgarry <https://github.com/chrislgarry> We have never been able to reproduce this issue in our testing environment. It would be great if you can provide a docker image that's able to reproduce the issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#149 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACGVQmT7tqg8hfLxcLUo71acqCjiV9omks5uIg68gaJpZM4DNciH> .

[ErichRitz]

I just hit the same issue: building failed, but running ./autogen.sh twice resulted in a successful build. The weird thing is I have build protobuf3 before and it built successfully. I'm using Slackware and build in a "clean" (full Slackware install with all patches installed) chroot environment using the SlackBuild file from https://slackbuilds.org/repository/14.2/misc/protobuf3/ . In both cases I built protobuf3 release version 3.5.2. So the only difference between my original successful build (May 22nd 2018) and my "need to run autogen.sh twice" build (March 6th 2019) is the Slackware patches I have installed.

A portion of the build log from the successful build:

+ autoreconf -f -i -Wall,no-obsolete
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
configure.ac:27: installing 'build-aux/compile'
configure.ac:30: installing 'build-aux/config.guess'
configure.ac:30: installing 'build-aux/config.sub'
configure.ac:24: installing 'build-aux/install-sh'
configure.ac:24: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
parallel-tests: installing 'build-aux/test-driver'
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: Consider adding 'AC_CONFIG_MACRO_DIRS([m4])' to configure.ac,
libtoolize: and rerunning libtoolize and aclocal.
libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
configure.ac:22: installing 'build-aux/compile'
configure.ac:25: installing 'build-aux/config.guess'
configure.ac:25: installing 'build-aux/config.sub'
configure.ac:19: installing 'build-aux/install-sh'
configure.ac:19: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
parallel-tests: installing 'build-aux/test-driver'
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
configure.ac:66: installing './ar-lib'
configure.ac:61: installing './compile'
configure.ac:46: installing './config.guess'
configure.ac:46: installing './config.sub'
configure.ac:48: installing './install-sh'
configure.ac:48: installing './missing'
benchmarks/Makefile.am: installing './depcomp'
parallel-tests: installing './test-driver'
+ rm -rf autom4te.cache config.h.in~
+ exit 0

A portion of the build log where I was forced to replace "./autogen.sh" with "./autogen.sh || ./autogen.sh" to get a successful build:

+ autoreconf -f -i -Wall,no-obsolete
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
configure.ac:27: installing 'build-aux/compile'
configure.ac:30: installing 'build-aux/config.guess'
configure.ac:30: installing 'build-aux/config.sub'
configure.ac:24: installing 'build-aux/install-sh'
configure.ac:24: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
parallel-tests: installing 'build-aux/test-driver'
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: Consider adding 'AC_CONFIG_MACRO_DIRS([m4])' to configure.ac,
libtoolize: and rerunning libtoolize and aclocal.
libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
configure.ac:22: installing 'build-aux/compile'
configure.ac:25: installing 'build-aux/config.guess'
configure.ac:25: installing 'build-aux/config.sub'
configure.ac:19: installing 'build-aux/install-sh'
configure.ac:19: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
parallel-tests: installing 'build-aux/test-driver'
libtoolize: putting auxiliary files in '..'.
libtoolize: copying file '../ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
configure.ac:66: installing './ar-lib'
configure.ac:61: installing './compile'
configure.ac:46: installing './config.guess'
configure.ac:46: installing './config.sub'
configure.ac:48: installing './install-sh'
configure.ac:94: error: required file './ltmain.sh' not found
configure.ac:48: installing './missing'
benchmarks/Makefile.am: installing './depcomp'
parallel-tests: installing './test-driver'
autoreconf: automake failed with exit status: 1
+ autoreconf -f -i -Wall,no-obsolete
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
configure.ac:27: installing 'build-aux/compile'
configure.ac:24: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: Consider adding 'AC_CONFIG_MACRO_DIRS([m4])' to configure.ac,
libtoolize: and rerunning libtoolize and aclocal.
libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
configure.ac:22: installing 'build-aux/compile'
configure.ac:19: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
configure.ac:61: installing './compile'
configure.ac:48: installing './missing'
benchmarks/Makefile.am: installing './depcomp'
+ rm -rf autom4te.cache config.h.in~
+ exit 0

Note the difference between the two trying to copy ltmain.sh:
Successful build:

libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'

autogen.sh twice build:

libtoolize: putting auxiliary files in '..'.
libtoolize: copying file '../ltmain.sh'
...
configure.ac:94: error: required file './ltmain.sh' not found
...
autoreconf: automake failed with exit status: 1
# run autogen.sh a second time
...
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'

So in a failing build it's looking for ltmain.sh in ".." for some reason but in a successful build it looks for ltmain.sh in "."

I have NOT tried reproducing on the latest stable version of protobuf3.

[ErichRitz]

Ok, I figured out why this is happening. This is what set me on the right track:
https://www.linuxquestions.org/questions/slackware-14/libtoolize-behavior-under-tmp-subdirectories-4175632753/

I (or rather, the build script) was extracting protobuf to /tmp/SBo/protobuf-3.5.2/ and building it there. A different program compilation had (incorrectly) left files directly in /tmp/SBo instead of a subdirectory, namely, /tmp/SBo/install.sh. libtoolize, which is called by autoreconf, which is called by ./autogen.sh during the build, was finding /tmp/SBo/install.sh and assuming /tmp/SBo was the proper build directory instead of /tmp/SBo/protobuf-3.5.2/ (or to be more specific, in relative path terms it was using ".." instead of ".".). Thus ltmain.sh was copied to the "wrong" location and the build failed. Removing /tmp/SBo/install.sh caused the build to be successful.

What I learned: The parent directory of your build directory should be clean of files, especially ones named install.sh. This is not a problem of protobuf, it is a problem of an unclean build environment. Hopefully this helps someone else!