Autoconf-2.68b wants testers

[2bits]

Not strictly an issue, but the folks at GNU are getting set to bump autoconf to 2.69. They have asked people to test autoconf-2.68b.

Your devel block for autoconf.rb would look like this:

  devel do
    url 'ftp://alpha.gnu.org/gnu/autoconf/autoconf-2.68b.tar.bz2'
    sha1 '8b6ca702e97ac51efa860cf959ca294ac4219856'
  end

and you probably want to ENV.deparallelize to reduce the variables before make check.
I posted this because now is the chance to get it right. Please brew rm -f autoconf and
repeat the test procedure at least twice. It takes 10min a run.

[jacknagel]

The mailing list is probably a more appropriate medium for something like this.

[2bits]

roger that. it's failing randomly for me, non-repeatable. If it can't do simple things, it will be an issue when someone pulls a 2.69. Can you just test it twice and close this with your results? thx. @samueljohn you might be interested too, as you probably have a clean test rig.

[jacknagel]

Running it now.

[2bits]

Thanks. They have a patch for test 100 already. So you can ignore that one.

[jacknagel]

Ran it twice, identical results:

ERROR: 461 tests were run,
5 failed (4 expected failures).
42 tests were skipped.

(it was test 100 that failed.)

[2bits]

Ahh, well i guess I'll wipe and reinstall. thx for the heads up.

[samueljohn]

have they changed the binary? I get and md5 hash of 592814f2deb0a5d4dc869ef420dc31f4. (where is my sha1 tool? Is it also missing if the CLTs are not installed?)

Besides that I get:

ERROR: 448 tests were run,
9 failed (4 expected failures).
55 tests were skipped.

This was run on my Xcode 4.3 without CLT branch.

[2bits]

Two ways to get your hashes:

brew fetch --devel autoconf
openssl dgst -sha1 /path/to/the/autoconf-2.68b.tar.bz2

You can also throw a --force in there to make it not reuse the cache version. I get the same hashes.

[2bits]

You should have had only 1 fail, test 100. Please retest a few times and see if it's consistent. Mine fails randomly, and I'm thinking it's my RAM, unless you get the same behavior.

[samueljohn]

The binary and the hash itself was fine.
The problem was because the non-devel md5 has was still used and not overwritten by the sha1 of the "do devel" part. I think this is a bug in homebrew.

In a formula like this:

class Autoconf < Formula
  homepage 'http://www.gnu.org/software/autoconf'
  url 'http://ftpmirror.gnu.org/autoconf/autoconf-2.68.tar.gz'
  mirror 'http://ftp.gnu.org/gnu/autoconf/autoconf-2.68.tar.gz'
  md5 'c3b5247592ce694f7097873aa07d66fe'

  devel do
    url 'ftp://alpha.gnu.org/gnu/autoconf/autoconf-2.68b.tar.bz2'
    sha1 '8b6ca702e97ac51efa860cf959ca294ac4219856'
  end

I get:

brew install autoconf --devel --fresh --force
==> Downloading ftp://alpha.gnu.org/gnu/autoconf/autoconf-2.68b.tar.bz2
######################################################################## 100.0%
Error: MD5 mismatch
Expected: c3b5247592ce694f7097873aa07d66fe
Got: 592814f2deb0a5d4dc869ef420dc31f4
Archive: /Users/sam/Library/Caches/Homebrew/autoconf-2.68b.tar.bz2

Besides that you seem right about the randomness.
For the second run I get:

ERROR: 448 tests were run,
10 failed (4 expected failures).
55 tests were skipped.

Now running that test on the machine with virgin 10.7.3 and CLT only (no Xcode).

[adamv]

When a devel block is used, both stable and devel need to use the same checksum type. This is kinda-sorta a known issue.

[mistydemeo]

Speaking of which, I will be fixing that (among other things).

[samueljohn]

Second run on my Xcode 4.3 (without CLTs):

ERROR: 448 tests were run,
8 failed (4 expected failures).
55 tests were skipped.

testsuite: 100 271 359 390 failed -> log

Third run:

ERROR: 448 tests were run,
10 failed (4 expected failures).
55 tests were skipped.

[GNU Autoconf 2.68b] testsuite: 100 271 359 412 451 465 failed -> log

So @2bits I guess it's not your RAM bat rather some multicore issue. Thinking of this, I forgot to set ENV.deparallelize. Will rerun again :-)
Further I have used my own experimental branch #10510.

On my MacMini with only the Command Line Tools for Xcode installed, I get inconsistent results, too:

Sometimes testsuite: 100 305 433 failed and other times: testsuite: 100 498 500 failed.

I mailed all reports to the suggested address.

[2bits]

Thanks. You did a great job. I would be curious what the results are with full XCode-4.3 + CLT. Do you have?
When a test is failed or skipped it leaves a directory behind. If you failed 100 302 458, then you would have these:

tests/testsuite.dir/100
tests/testsuite.dir/302
tests/testsuite.dir/458

After dozens of runs generating between 2 or 10 fails, I still can't define the problem, but I found a commonality, they almost always seem to complain that there's a diff comparing environment before and after. You will see lots of .before and .after files that you can tkdiff.

[samueljohn]

@2bits I got a response from the autoconf makers. We should test their latest autoconf.git version. I tried to hack it into the formula but I failed. They ask if test Nr 100 (and perhaps two others) are fixed in the latest version.
Do you think, you could just hack the autoconf.rb formula such that we can brew install -v -d autoconf --HEAD for testing?
That would be great!

And I don't have Xcode-4.3 + CLT. Sorry. I wish I had another Mac. Once in a while I can test on the MacBook of my girlfriend (where I installed both g).

[2bits]

Testing HEAD is next to impossible. I've been trying it for two weeks. You need a lot of deps including autoconf itself, and it will build, but when it comes to the testsuite, it complains that autoconf-2.68 is older than 2.57 and bails. I never got a response to my requests for help on that. It will take too much of your time and 2.68b is a better testbed. It just runs.

The patches they offered are good patches. I've tried them before. Thy fix 100, but have zero affect on randomness. What I can tell you is that every single test has passed at one point or another, and if you retest a test, it works. So don't think anything but 100 is broken. It's the testsuite and their assumptions on how OSX works that's questionable, because those assumptions could spill out into 2.69 and make it randomly fail.

I will be doing clean install of Lion + full XCode + CLT today. It's important I get a good baseline once a month if I'm writing formulas anyway. Really appreciate your help on this.

[samueljohn]

@2bits oh I see :-) Then I'll try again with the current 2.68b.
Btw. ENV.deparallelize doesn't change my results.
So I hope having these failures is not the consequence of my "Xcode without CLT"- attempt. Perhaps I can persuade you to try out #10510 with your new full Xcode + CLT?

[2bits]

I made a branch and folded with all their patches. You can see what I did here.

To confirm HB had nothing to do with it, you can brew unpack it and build it without setting CC or CXX or anything.
I'm happy to fork your repo or brew pull that request, a bit later, once I see if these fails happen to XCode+CLT.

[samueljohn]

@2bits Ah, good work! The test No 100 is fixed for me but other (random) remain.

[GNU Autoconf 2.68b] testsuite: 271 359 367 387 481 493 failed
[GNU Autoconf 2.68b] testsuite: 271 359 399 failed
[GNU Autoconf 2.68b] testsuite: 271 352 359 485 failed

These two seem to fail always on my core i7 sandy bridge (SSD) with Xcode but not CLT:

Low level compiling/preprocessing macros.
271: Multiple languages                              FAILED (compile.at:463)
Semantics.
359: AC_CHECK_HEADERS                                FAILED (semantics.at:228)

From Paul Eggert's Mail in response to my bug report:

Thanks, could you please try it again with the following patches,
and let us know how that works?

http://lists.gnu.org/archive/html/bug-autoconf/2012-03/txtmjhqa66pjL.txt

This should fix at least test 100, and I hope it fixes the other two.

@2bits do you think you could incorporate these other patches into you branch?

[2bits]

All the patches are in there I think, but I'll double check. You can easily run test 359 again with ./testsuite 359. You should prove to yourself it consistently fails, as I have passed every single test at some point or another. It's good news to hear you can get a repeat fail though. My curiosity increases...

[samueljohn]

Ok, @2bits I'll do so and report back later. Thank you.

[2bits]

The problem has been identified as a bug in Apple's compile of grep-2.5.1 on Lion, where grep -E fails randomly when the regex is formatted a certain way and returns an error:

Regular expression too big

causing these autoconf tests of our environment to fail. The technical analysis was done last May on bug.mozilla. You can read about it here As of a few days ago, [ANNOUNCE] git-1.7.9.3 email shows them discussing this issue, and trying to define it.

@jacknagel I suppose the course of action is to noop the grep issue, as we aren't going to dupe it to fix it. Does Apple have a public bugtracker? I haven't found one.

[jacknagel]

Heh, I have a newer grep in my PATH so I didn't experience the failure.

Yeah, I mean if the autoconf folks want to maintain compatibility with OS X with minimal dependencies then they're going to have to skip that test or something. Even if we did dupe grep (not suggesting this), that doesn't help people building it manually or otherwise.