`cabal install haskell-src-exts-1.10.1` indeterministically fails to regenerate the happy parser #2311

hvr · 2015-01-02T19:20:44Z

I noticed this because with GHC 7.8.4 cabal install haskell-src-exts-1.10.1 would sometimes succeed and sometimes fail with a compile failure in InternalParser.hs, depending on whether it was regenerated or not (GHC 7.8.4 requires parsers created by a recent enough happy)

I tracked this down to the way source-tarball are extracted, which seem to ignore the timestamps recored in the tarballs, and instead extracts the file with current system time.

So here's two example runs (I looked at the timestamps before cabal would decide whether to preprocess any files), which result in two cases in which the two files are in a different relative temporal relationship to each other:

case 1: .hs file older than .ly file (.hs file will be regenerated)

-rw-rw-r-- 1 hvr hvr 505437 2015-01-02 19:59:46.556436392 +0100 haskell-src-exts-1.10.1-30225/haskell-src-exts-1.10.1/dist/dist-sandbox-4ac2267c/build/Language/Haskell/Exts/InternalParser.hs
-rw-rw-r-- 1 hvr hvr  91522 2015-01-02 19:59:46.560436366 +0100 haskell-src-exts-1.10.1-30225/haskell-src-exts-1.10.1/src/Language/Haskell/Exts/InternalParser.ly

case 2: .hs exact same age as .ly file (.hs file is not regenerated -> compile failure)

-rw-rw-r-- 1 hvr hvr 505437 2015-01-02 20:01:46.371660280 +0100 haskell-src-exts-1.10.1-31835/haskell-src-exts-1.10.1/dist/dist-sandbox-4ac2267c/build/Language/Haskell/Exts/InternalParser.hs
-rw-rw-r-- 1 hvr hvr  91522 2015-01-02 20:01:46.371660280 +0100 haskell-src-exts-1.10.1-31835/haskell-src-exts-1.10.1/src/Language/Haskell/Exts/InternalParser.ly

Depending on whether case 1 or 2 occured (which depends on the operating system's clock accuracy, and interaction with process scheduling), the parser would be either regenerated or not.

There are a couple ways to workaround/fix this issue, here's just some ideas:

A simple fix which would work in this case is to change the logic which decides whether to preprocess parsers to also regenerate in case of same-timestamps (this works as long as .ly is always extracted before the .hs -- does cabal sdist enforce this ordering in the tarball?)
cabal install some-package could always consider a .ly file to be regenerated, because of GHC 7.8.x and later requiring recent enough happy to have generated the parser
Extract files from source-tarball with recorded time-stamp. This would at least make the situation deterministic, although the parser would then probably never be regenerated, as the generated .hs file's timestamp would normally away be younger than the .ly file's.

/cc @feuerbach

The text was updated successfully, but these errors were encountered:

dcoutts · 2015-01-06T10:05:31Z

Well done tracking it down to the tarball timestamps.

dcoutts · 2015-01-06T10:17:06Z

Using the timestamps when unpacking seems like a sensible thing to do.

The issue with ghc-7.8 and older happy should be dealt with separately as a special case.

This changes `moreRecentFile a b` to return true not only when `a` is younger than `b`, but also when `a` is exactly the same age of `b`, as that case is subject to race-conditions, and it's better to err on assuming it needs to be regenerated. This is an attempt to provide a pragmatic workaround for haskell#2311

23Skidoo · 2015-01-14T16:22:37Z

Related: #130.

This adds `notLessRecentFile a b` to return true not only when `a` is younger than `b`, but also when `a` is exactly the same age of `b`, as that case is subject to race-conditions (if the system time granularity is too low), and it's better to err on assuming it needs to be regenerated. This is an attempt to provide a pragmatic workaround for haskell#2311

NicolasT · 2015-12-01T23:50:50Z

I'm somewhat bitten by this as well: my package contains an in-tree autotooled library. A package sdist tarball contains correct timestamps: configure.ac and aclocal.m4 are 'older' than configure, config.h.in etc.
Due to Cabal not honouring timestamps during unpack, when running make after configure (done through a Custom Setup.hs and such), the autotools 'maintainer mode' rules kick in, after which on some systems (including Hackage) the build bails out because aclocal etc are missing (and they shouldn't be required either).

So, as long as cabal unpack doesn't honour tarball timestamps, I need to work-around this somehow (e.g. by explicitly passing --disable-maintainer-mode to configure, which isn't considered OK nowadays).

After investigation of the build failures on Hackage, caused by autools 'maintainer mode' kicking in and required packages not being installed on the build systems, it turns out `cabal unpack` doesn't honor timestamps of files contained in `sdist` tarballs while unpacking. This in turn causes some source files in `cbits` to appear newer than their generated versions (e.g. `configure.ac` being newer than `configure`), and the 'maintainer mode' rules being triggered, which we obviously don't want to happen in release packages. This patch changes `Setup.hs` to explicitly pass `--disable-maintainer-mode` to `configure` in builds of packages whose version number doesn't match the development version, 999. This is a work-around for Cabal issue #2311. See: haskell/cabal#2311

Release 0.0.1.0 with changes to `Setup.hs` to pass `--disable-maintainer-mode` to `configure` in release builds After investigation of the build failures on Hackage, caused by autools 'maintainer mode' kicking in and required packages not being installed on the build systems, it turns out `cabal unpack` doesn't honor timestamps of files contained in `sdist` tarballs while unpacking. This in turn causes some source files in `cbits` to appear newer than their generated versions (e.g. `configure.ac` being newer than `configure`), and the 'maintainer mode' rules being triggered, which we obviously don't want to happen in release packages. This patch changes `Setup.hs` to explicitly pass `--disable-maintainer-mode` to `configure` in builds of packages whose version number doesn't match the development version, 999. This is a work-around for Cabal issue #2311. See: haskell/cabal#2311 See: 4eeceb8 * tag 'reedsolomon-0.0.1.2': Release: Add generated autotools files Release: Set version number

Release 0.0.1.1 with disabled maintainer mode during `cbits` build After investigation of the build failures on Hackage, caused by autools 'maintainer mode' kicking in and required packages not being installed on the build systems, it turns out `cabal unpack` doesn't honor timestamps of files contained in `sdist` tarballs while unpacking. This in turn causes some source files in `cbits` to appear newer than their generated versions (e.g. `configure.ac` being newer than `configure`), and the 'maintainer mode' rules being triggered, which we obviously don't want to happen in release packages. This patch changes `Setup.hs` to explicitly pass `--disable-maintainer-mode` to `configure` in builds of packages whose version number doesn't match the development version, 999. This is a work-around for Cabal issue #2311. See: haskell/cabal#2311 See: 4eeceb8 * tag 'reedsolomon-0.0.1.2': Release: Add generated autotools files Release: Set version number

hvr · 2016-05-19T15:39:46Z

Fwiw, since directory-1.2.3 there's now a portable setModificationTime operation (see also haskell/directory#13). So cabal unpack could now try when compiled against directory >= 1.2.3 to set the mtime (and tolerate failure)?

hvr · 2016-06-19T14:56:10Z

After some more investigation, it turns out that some time ago haskell/tar@761bbc4 landed, so starting with tar-0.5.0.0 (in combination with directory >= 1.2.3), cabal sets modification times when unpacking source-tarballs.
So this should make cabal behave more deterministically.

As to whether this issue is fixed: depends on the versions of tar and directory that were used for building cabal :-/

23Skidoo · 2016-06-19T19:00:05Z

Can we require tar >= 0.5 && directory >= 1.2.3 for 1.26?

gbaz · 2018-02-07T09:00:29Z

The tar dep is bumped, but the directory dep (for cabal install) is still 1.2.2. Can that be bumped?

hvr · 2018-02-07T09:03:34Z

Note that GHC 7.10.3 had directory-1.2.2 bundled, while GHC 8.0.1 had directory-1.2.6.2 bundled

gbaz · 2018-02-07T21:56:58Z

i forget the rules for ghc support windows..

gbaz · 2018-02-14T19:03:00Z

Hrm -- 7.10.3 was released 8th December 2015, so by the three year rule it needs to be supported until december 2018. That said -- it is allowable to bootstrap cabal-install (rather than cabal-the-library) with newer versions of libs than those shipped with ghc, so in theory it sounds like we could bump the version here if the bootstrap could be confirmed to work?

Or we could just wait a year, and then bump the dep with no work at all :-P

hvr added type: bug cabal-install: other labels Jan 2, 2015

hvr added this to the Cabal-1.22 milestone Jan 2, 2015

hvr mentioned this issue Jan 14, 2015

Change moreRecentFile test to workaround races #2362

Closed

ttuegel modified the milestones: Cabal-1.24, Cabal-1.22 Apr 23, 2015

hvr mentioned this issue Dec 24, 2015

cabal sdist should touch preprocessed .hs files #2940

Closed

23Skidoo modified the milestones: Cabal 1.24, Cabal 1.26 Feb 21, 2016

ezyang modified the milestone: Cabal 2.0 Sep 6, 2016

judah mentioned this issue Jan 3, 2017

Make the bootstrapping process more robust: google/proto-lens#66

Merged

judah mentioned this issue Jan 29, 2017

Include generated files in releases of packages depending on proto-lens google/proto-lens#69

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`cabal install haskell-src-exts-1.10.1` indeterministically fails to regenerate the happy parser #2311

`cabal install haskell-src-exts-1.10.1` indeterministically fails to regenerate the happy parser #2311

hvr commented Jan 2, 2015

dcoutts commented Jan 6, 2015

dcoutts commented Jan 6, 2015

23Skidoo commented Jan 14, 2015

NicolasT commented Dec 1, 2015

hvr commented May 19, 2016 •

edited

Loading

hvr commented Jun 19, 2016 •

edited by 23Skidoo

Loading

23Skidoo commented Jun 19, 2016

gbaz commented Feb 7, 2018

hvr commented Feb 7, 2018

gbaz commented Feb 7, 2018

gbaz commented Feb 14, 2018

cabal install haskell-src-exts-1.10.1 indeterministically fails to regenerate the happy parser #2311

cabal install haskell-src-exts-1.10.1 indeterministically fails to regenerate the happy parser #2311

Comments

hvr commented Jan 2, 2015

dcoutts commented Jan 6, 2015

dcoutts commented Jan 6, 2015

23Skidoo commented Jan 14, 2015

NicolasT commented Dec 1, 2015

hvr commented May 19, 2016 • edited Loading

hvr commented Jun 19, 2016 • edited by 23Skidoo Loading

23Skidoo commented Jun 19, 2016

gbaz commented Feb 7, 2018

hvr commented Feb 7, 2018

gbaz commented Feb 7, 2018

gbaz commented Feb 14, 2018

`cabal install haskell-src-exts-1.10.1` indeterministically fails to regenerate the happy parser #2311

`cabal install haskell-src-exts-1.10.1` indeterministically fails to regenerate the happy parser #2311

hvr commented May 19, 2016 •

edited

Loading

hvr commented Jun 19, 2016 •

edited by 23Skidoo

Loading