Permalink
Commits on Sep 25, 2016
  1. Use hash-consing to optimise index cache (#3897)

    Without this optimisation, `cabal info somethingnonexisting` results in
    
         960,397,120 bytes allocated in the heap
         739,652,560 bytes copied during GC
          67,757,128 bytes maximum residency (24 sample(s))
           2,234,096 bytes maximum slop
                 147 MB total memory in use (0 MB lost due to fragmentation)
    
    with this optimisation:
    
       1,000,825,744 bytes allocated in the heap
         656,112,432 bytes copied during GC
          44,476,616 bytes maximum residency (24 sample(s))
           2,302,864 bytes maximum slop
                 109 MB total memory in use (0 MB lost due to fragmentation)
    
    So the total memory in use is significantly lower. The total runtime is
    also slightly reduced, from
    
      INIT    time    0.001s  (  0.001s elapsed)
      MUT     time    0.683s  (  1.050s elapsed)
      GC      time    0.946s  (  0.946s elapsed)
      EXIT    time    0.005s  (  0.005s elapsed)
      Total   time    1.637s  (  2.002s elapsed)
    
    to
    
      INIT    time    0.001s  (  0.001s elapsed)
      MUT     time    0.664s  (  0.988s elapsed)
      GC      time    0.797s  (  0.797s elapsed)
      EXIT    time    0.004s  (  0.004s elapsed)
      Total   time    1.467s  (  1.789s elapsed)
    
    
    Note that there's currently ~80k cache entries, but only ~10k unique package names
    and ~6k unique versions. So hash-consing helps reduce the amount of heap objects 
    for both value types by one order of magnitude, which among other benefits also
    reduces GC overhead.
    hvr committed on GitHub Sep 25, 2016
Commits on Sep 24, 2016
  1. Merge pull request #3894 from hvr/pr/rtsopts

    Enable `-rtsopts` for `cabal` executable
    23Skidoo committed on GitHub Sep 24, 2016
  2. Merge PR #3893 (Implement `--index-state` aka index freezing)

    This PR implements a new flag `--index-state` (and its `cabal.project`/config-file
    equivalent `index-state: ...`) which allows to change the source package index
    state the solver uses to compute install-plans. This is particularly useful in
    combination with freeze-files in order to also freeze the state the package
    index was in at the time the install-plan was frozen.
    
    This provides the core functionality, on which future enhancements can build
    upon. See also description of PR #3604 for some possible enhancements.
    hvr committed on GitHub Sep 24, 2016
  3. Initial minimal documentation for `--index-state`

    [skip ci]
    hvr committed Sep 24, 2016
  4. Enable `-rtsopts` for `cabal` executable

    Fwiw, the GHC executables are compiled with full `-rtsopts` as well.
    The benefit is that one can use flags such as `-M1G` to limit the heap
    space, or profile memory usage via `-hT`.
    hvr committed Sep 24, 2016
  5. Turn 'Timestamp' into a proper type (#3892)

    Most notably, this allows us to provide a custom instance for `Text`
    hvr committed on GitHub Sep 24, 2016
Commits on Sep 22, 2016
  1. Grammar.

    23Skidoo committed Sep 22, 2016
  2. Merge pull request #3884 from 23Skidoo/print-num-jobs

    Print the number of jobs we're using.
    23Skidoo committed on GitHub Sep 22, 2016
  3. Print the number of jobs we're using.

    As requested by @hvr.
    23Skidoo committed Sep 22, 2016
  4. Fix compile error for Windows build

    Turns out that `RankNTypes` is only needed for the windows-conditional
    code parts in `CreatePipe`
    hvr committed Sep 22, 2016
  5. Update default-language & avoid default-extensions (#3880)

    This upgrades the `default-language: Haskell98` to `Haskell2010`
    and removes `default-extensions: RankNTypes, FlexibleContexts`
    in favor of adding `LANGUAGE` pragmas where needed.
    
    Moroever, this also drops `LANGUAGE` pragmas which have become redundant
    due to `Haskell2010` (specifically, `EmptyDataDecls`,
    `ForeignFunctionInterface` and `PatternGuards`)
    
    Finally, an `other-extensions` specification is put in place for the
    `Cabal` library component.
    
    This helps loading up files directly in GHCi, such as e.g. `ghci Setup.hs`
    without having to specify `-X...` flags.
    hvr committed on GitHub Sep 22, 2016
Commits on Sep 21, 2016
  1. Merge pull request #3879 from ezyang/pr/no-legacy-exe-solving

    Don't solve for executables in legacy code path.
    23Skidoo committed on GitHub Sep 21, 2016
  2. Merge pull request #3874 from 23Skidoo/user-config-init-remote-repo

    'user-config init': Print the default 'remote-repo' correctly.
    23Skidoo committed on GitHub Sep 21, 2016
  3. Don't solve for executables in legacy code path.

    There is a bug in `cabal configure`'s invocation of the solver in
    Distribution/Client/Configure.hs:
    
        standardInstallPolicy
            installedPkgIndex
            (SourcePackageDb mempty packagePrefs)
            [SpecificSourcePackage localPkg]
    
    We can see that the solver is given an EMPTY source package database.
    This is because we assume that everything you need from cabal configure
    is taken from the installed package index.
    
    But this is NOT true for executables, which never have an entry in the
    installed package index. So we SHOULD NOT solve for
    executables in the legacy codepath, since there isn't anything useful we
    can do with the info anyway.  This gets toggled using a new solver
    parameter SolveExecutables.
    
    I didn't bother with a test because this will be obsoleted by
    nix-local-build.
    
    Fixes #3875
    
    Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>
    ezyang committed Sep 21, 2016
  4. Merge pull request #3873 from ezyang/pr/nix-local-build-prof

    Never use --enable-profiling when invoking Setup.
    ezyang committed on GitHub Sep 21, 2016
  5. Merge pull request #3878 from hvr/pr/extended-01-index-cache

    - Extend `01-index.cache` with timestamps in preparation of `--index-state`
    - Use 'Binary' encoding for `01-index.cache`
    hvr committed on GitHub Sep 21, 2016
  6. Refactor & optimise construction of index cache

    This commit was motivated by @dcoutts' code-review comment:
    
    > Originally with using the `Sec.directoryEntries` that gave us only the
    > final version of each file, ie not all intermediate revisions. And
    > previously our strategy was to go through the final versions of each
    > file, in file order, and lookup just the ones we're interested in (which
    > in practice is 99% of them).
    >
    > Now for the new cache we want to go through all revisions, which means
    > all entries in file order. So instead of using `Sec.directoryEntries`
    > which reads from the tar index, we go straight for `Sec.directoryFirst`
    > which is block 0 and iterate through, using `lazyUnfold`.
    >
    > But we can now significantly simplify this and do it more
    > efficiently. Note that `indexLookupEntry` and `indexLookupFileEntry` are
    > expensive operations that seek in the tar file and read the tar entry at
    > that point. So lets do it exactly once per entry. The current code does
    > it once in the `lazyUnfold indexLookupEntry` and then again in `mk`. But
    > the old `mk` only did that because it had not previously looked up the
    > entry.
    hvr committed Sep 21, 2016
Commits on Sep 20, 2016
  1. Merge pull request #3877 from phadej/travis-branches

    Whitelist branches for travis to build
    23Skidoo committed on GitHub Sep 20, 2016
  2. Try to regenerate a corrupted 01-index.cache

    With this commit, if a corrupted index cache is detected the
    `readIndexCache` function now regenerates the index cache and then
    reattempt to read the index once (and 'die's if it fails again).
    hvr committed Sep 20, 2016
  3. Extend 01-index.cache & use 'Binary' encoding

    This commit extends the index cache entries relevant for 01-index to
    include block numbers and timestamps, and makes them strict so recent
    GHCs unpack the fields:
    
        data IndexCacheEntry
            = CachePackageId PackageId BlockNo
            | CachePreference Dependency
            | CacheBuildTreeRef BuildTreeRefType BlockNo
    
    to
    
       data IndexCacheEntry
           = CachePackageId PackageId !BlockNo !Timestamp
           | CachePreference Dependency !BlockNo !Timestamp
           | CacheBuildTreeRef !BuildTreeRefType !BlockNo
    
    For the legacy `00-index.tar`s, the 'Timestamp' field is set to (-1),
    and the original 00-index.cache format is retained.
    
    For (secure) `01-index.tar`s, all of `IndexCacheEntry`s data is stored
    in the `01-index.cache` file.
    
    Moreover, to avoid having to write out and parse new two integers per
    cache entry, this patch switches to using `Binary` instances for
    encoding the `01-index.cache` file (while `00-index.cache` remains
    plain-text).
    hvr committed Sep 19, 2016
  4. Merge pull request #3876 from hvr/pr/genericRnf

    Add helper for Generic NFData instances
    23Skidoo committed on GitHub Sep 20, 2016
  5. Revert "Merge pull request #3863 from haskell/installplan-installed-s…

    …tate"
    
    This reverts commit 7b67f37, reversing
    changes made to c12290f.
    ezyang committed Sep 20, 2016
  6. Merge pull request #3863 from haskell/installplan-installed-state

    Installplan installed state
    ezyang committed on GitHub Sep 20, 2016
  7. Add `Cabal.Distribution.Compat.Prelude.genericRnf`

    This is needed in order to support `deepseq < 1.4` which didn't
    have a `Generic`-based default `rnf` implementation yet.
    
    In order to define instances, use e.g.
    
    > instance NFData MyType where rnf = genericRnf
    
    The implementation has been taken from `deepseq-1.4.2`'s default
    `rnf` implementation.
    hvr committed Sep 20, 2016
  8. Merge pull request #3868 from ezyang/pr/build-msg-custom

    Print that we are building all due to Custom setup.
    23Skidoo committed on GitHub Sep 20, 2016
  9. Merge pull request #3869 from ezyang/pr/minor-renaming-config-prof

    Rename configProf/configCoverage local variables to not shadow.
    23Skidoo committed on GitHub Sep 20, 2016
  10. Merge pull request #3870 from dcoutts/getopt-fork-tidy

    Clarify the differences with the GetOpt from base
    BardurArantsson committed on GitHub Sep 20, 2016
Commits on Sep 19, 2016
  1. Never use --enable-profiling when invoking Setup.

    In Cabal 1.22.5.0, the semantics of
    --disable-profiling/--enable-profiling depend on ordering (because there
    is a hack that operates by looking at the current flag assignment and
    doing something). In particular, if I specify --enable-library-profiling
    --disable-profiling, I end up with library profiling DISABLED.
    
    The fix is that we NEVER pass --enable-profiling or --disable-profiling
    to Cabal. At the moment, and according to my historical analysis, Cabal
    ONLY uses configProf to affect the effective library/executable
    profiling, which means that anything we do with --enable-profiling, we
    can do using the library/executable profiling individually. Since these
    are always flags for the versions of Cabal library we support, we will
    get order invariance. Historical versions have varied on whether or not
    setting executable profiling implies library profiling, but if we set
    both explicitly this change in behavior doesn't matter.
    
    This patch is difficult to test because the bad profiling flags
    can't be induced on an inplace build.  I tested by hand by building
    a package that depended on 'distributive' by hand.
    
    Fixes #3790.
    
    Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>
    ezyang committed Sep 19, 2016