Merge remote-tracking branch 'origin/stable' into stable-into-master

ChangeLog.md

@@ -50,6 +50,9 @@ Bug fixes:
   Previously, if you SIGTERMed at the wrong time while running a script, you
   could end up with an inconsistent database state.
 
+* `--resolver global` doesn't retrieve snapshots list from the internet
+   beause doesn't need it. See [#5103](https://github.com/commercialhaskell/stack/issues/5103)
+
 * Fix using relative links in haddocks output.  See
   [#4971](https://github.com/commercialhaskell/stack/issues/4971).
 * Do not include generated cabal file information in lock files. See

doc/GUIDE.md

@@ -1825,13 +1825,11 @@ To generate a backtrace in case of exceptions during a test or benchmarks run,
 use the `--trace` flag. Like `--profile` this compiles with profiling options,
 but adds the `+RTS -xc` runtime option.
 
-### DWARF
+### Debugging symbols
 
-`stack` now supports debugging and profiling with
-[DWARF information](https://ghc.haskell.org/trac/ghc/wiki/DWARF),
-using the `--no-strip`, `--no-library-stripping`, and `--no-executable-stripping`
-flags to disable the default behavior of removing such information from compiled
-libraries and executables.
+Building with debugging symbols in the [DWARF information](https://ghc.haskell.org/trac/ghc/wiki/DWARF) is supported by `stack`. This can be done by passing the flag `--ghc-options="-g"` and also to override the default behaviour of stripping executables of debugging symbols by passing either one of the following flags: `--no-strip`, `--no-library-stripping` or `--no-executable-stripping`.
+
+In Windows GDB can be isntalled to debug an executable with `stack exec -- pacman -S gdb`. Windows visual studio compiler's debugging format PDB is not supported at the moment. This might be possible by [separating](https://stackoverflow.com/questions/866721/how-to-generate-gcc-debug-symbol-outside-the-build-target) debugging symbols and [converting](https://github.com/rainers/cv2pdb) their format. Or as an option when [using the LLVM backend](http://blog.llvm.org/2017/08/llvm-on-windows-now-supports-pdb-debug.html).
 
 ## More resources
 

doc/azure_ci.md

@@ -15,9 +15,7 @@ need to put the relevant configuration files:
   [azure-simple](https://raw.githubusercontent.com/commercialhaskell/stack/stable/doc/azure/azure-simple.yml)
   file into `azure-pipelines.yml`.
 * For complex Azure configuration, you need to take the below linked
-  four files and put all of them into the `.azure` directory except
-  the `azure-pipelines.yml` file which should be put in the root of
-  the repository.
+  four files and put all of them into the `.azure` directory.
 
 For a more detailed explanation, you can read further.
 

doc/coverage.md

@@ -81,7 +81,7 @@ However, advanced users may want to understand exactly how `--coverage` works:
    the executable.
 
    When switching on this flag, it will usually cause all local packages to be
-   rebuilt (see [#1940](https://github.com/commercialhaskell/stack/issues/1940).
+   rebuilt (see [#1940](https://github.com/commercialhaskell/stack/issues/1940)).
 
 2. Before the build runs with `--coverage`, the contents of `stack path --local-hpc-root`
    gets deleted. This prevents old reports from getting mixed

doc/faq.md

@@ -581,3 +581,19 @@ It's possible to bypass the warning for a specific version of GHC by modifying a
 **Note that we're fixing `ghc-8.2.2` in this case; repeat for other versions as necessary.** You should apply this fix for the version of GHC that matches your resolver.
 
 Issue [#4009](https://github.com/commercialhaskell/stack/issues/4009) on GitHub goes into further detail.
+
+## How do I install ghc in stack when it fails with the error: Missing ghc bindist for "linux64-ncurses6"?
+
+Example Error:
+```
+No setup information found for ghc-8.6.4 on your platform.
+This probably means a GHC bindist has not yet been added for OS key 'linux64-ncurses6'.
+Supported versions: ghc-7.10.3, ghc-8.0.1, ghc-8.0.2, ghc-8.2.1, ghc-8.2.2
+```
+
+Most Linux distributions have standardized on providing libtinfo.so.6 (either directly or as a symlink to libncursesw.so.6). As such, there aren't GHC 8.6.* bindists that link to libncursesw.so.6 available. 
+
+So creating a symlink to libncursesw.so.6 as libtinfo.so.6 can prevent this error (root privileges might be required).
+```
+ln -s /usr/lib/libncursesw.so.6 /usr/lib/libtinfo.so.6
+```

doc/install_and_upgrade.md

@@ -313,7 +313,7 @@ For more information and other shells, see [the shell auto-completion page](shel
 
 If you're attempting to install stack from within China:
 
-* As of 2018-10-24, the download link has limited connectivity from within mainland China. If this is the case, please proceed by manually downloading (ideally via a VPN) and installing stack per the instructions found on this page pertinent to your OS.
+* As of 2020-02-24, the download link has limited connectivity from within mainland China. If this is the case, please proceed by manually downloading (ideally via a VPN) and installing stack per the instructions found on this page pertinent to your OS.
 
 * After install, your ~/.stack/config.yaml will need to be configured before stack can download large files consistently from within China (without reliance on a VPN). Please add the following to the bottom of the ~/.stack/config.yaml file (for Windows: use the %STACK_ROOT%\config.yaml):
 
@@ -325,9 +325,20 @@ urls:
   latest-snapshot: http://mirrors.tuna.tsinghua.edu.cn/stackage/snapshots.json
 
 package-indices:
- - name: Tsinghua
-   download-prefix: http://mirrors.tuna.tsinghua.edu.cn/hackage/package/
-   http: http://mirrors.tuna.tsinghua.edu.cn/hackage/00-index.tar.gz
+  - download-prefix: http://mirrors.tuna.tsinghua.edu.cn/hackage/
+    hackage-security:
+        keyids:
+        - 0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d
+        - 1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42
+        - 280b10153a522681163658cb49f632cde3f38d768b736ddbc901d99a1a772833
+        - 2a96b1889dc221c17296fcc2bb34b908ca9734376f0f361660200935916ef201
+        - 2c6c3627bd6c982990239487f1abd02e08a02e6cf16edb105a8012d444d870c3
+        - 51f0161b906011b52c6613376b1ae937670da69322113a246a09f807c62f6921
+        - 772e9f4c7db33d251d5c6e357199c819e569d130857dc225549b40845ff0890d
+        - aa315286e6ad281ad61182235533c41e806e5a787e0b6d1e7eef3f09d137d2e9
+        - fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0
+        key-threshold: 3
+        ignore-expiry: no
 ```
 
 ## Using an http proxy
@@ -358,3 +369,25 @@ There are essentially four different approaches to upgrade:
       wget -qO- https://get.haskellstack.org/ | sh -s - -f
 
 * Manually follow the steps above to download the newest binaries from the release page and replace the old binary.
+
+## Install Older Versions
+
+To install a specific version of stack, navigate to the desired version on 
+[the GitHub release page](https://github.com/fpco/stack/releases),
+and click the appropriate link under its "Assets" drop-down menu.
+
+Alternatively, use the URL 
+`https://github.com/commercialhaskell/stack/releases/download/vVERSION/stack-VERSION-PLATFORM.EXTENSION`.
+For example, the tarball for stack 2.1.0.1, osx-x86_64 is at
+`https://github.com/commercialhaskell/stack/releases/download/v2.1.0.1/stack-2.1.0.1-osx-x86_64.tar.gz`.
+
+Here's a snippet for `appveyor.yml` files, borrowed from `dhall`'s 
+[`appveyor.yml`](https://github.com/dhall-lang/dhall-haskell/blob/1079b7a3a7a6922f72a373e47daf6f1b74f128b1/appveyor.yml).
+Change the values of PATH and VERSION as needed.
+
+    install:
+      - set PATH=C:\Program Files\Git\mingw64\bin;%PATH%
+      - curl --silent --show-error --output stack.zip --location "https://github.com/commercialhaskell/stack/releases/download/v%STACK_VERSION%/stack-%STACK_VERSION%-windows-x86_64.zip"
+      - 7z x stack.zip stack.exe
+      - stack setup > nul
+      - git submodule update --init --recursive

doc/maintainers/docker.md

@@ -0,0 +1,91 @@
+# Docker images
+
+Each Stackage LTS release has two corresponding docker images in the [fpco/stack-build](https://hub.docker.com/r/fpco/stack-build/) and [fpco/stack-build-small](https://hub.docker.com/r/fpco/stack-build-small/) repositories.  The former contains every system library needed to build any package in the snapshot, while the latter only contains a minimal set of system libraries for basic programs.
+
+The Dockerfiles for building these images are in [stackage/automated/dockerfiles](https://github.com/commercialhaskell/stackage/tree/master/automated/dockerfiles/).  There is also a [build.sh](https://github.com/commercialhaskell/stackage/tree/master/automated/dockerfiles/build.sh) script to help with building and pushing the images (see the [README](https://github.com/commercialhaskell/stackage/tree/master/automated/dockerfiles/README.md) for usage instructions).
+
+
+## Build images for new minor LTS snapshot
+
+In most cases, a new minor LTS snapshot just needs the previous LTS image to be re-tagged and pushed.  If the image needs a patch for the new minor LTS snapshot, see the next section.
+
+Below, replace `<N>.<M>` with the minor LTS snapshot version.
+
+- Check out the `stable` branch of the [stack repo](https://github.com/commercialhaskell/stack/).
+
+- Build and push the images (both standard and `small` variants) using the [build.sh](https://github.com/commercialhaskell/stackage/tree/master/automated/dockerfiles/build.sh) script: `./build.sh --push lts-<N>.<M> && ./build.sh --push --small lts-<N>.<M>`.
+
+
+## Patch images for new minor LTS snapshot
+
+Below, replace `<N>.<M>` with the minor LTS snapshot version. and `<N>.<M-1>` with the previous minor LTS snapshot version.
+
+- Check out the `stable` branch of the [stack repo](https://github.com/commercialhaskell/stack/).
+
+- In `stackage/automated/dockerfiles`, create a new `lts-<N>.<M>` directory.
+
+- Create `lts-<N>.<M>/Dockerfile`, starting with:
+
+        FROM $DOCKER_REPO:lts-<N>.<M-1>
+
+- Add layers for any changes that need to be made to the image.
+
+- Build the new image using the [build.sh](https://github.com/commercialhaskell/stackage/tree/master/automated/dockerfiles/build.sh) script: `./build.sh lts-<N>.<M> && ./build.sh --small lts-<N>.<M>`
+
+- Test the new image.  For example, `(stack --resolver=lts-<N>.<M> new image-test && cd image-test && stack --docker build)` (this should use the image you just built).  Make sure you test that the new image actually contains the desired changes.
+
+- Follow the process in the previous section to push the images.
+
+
+## Build images for new major LTS snapshot release
+
+### Test a Dockerfile prior to new major LTS snapshot release
+
+Replace `<N>` with major version of new LTS snapshot, and `<N-1>` with previous major LTS snapshot version.
+
+- Check out the `stable` branch of the [stack repo](https://github.com/commercialhaskell/stack/).
+
+- In `stackage/automated/dockerfiles`, create a new `lts-<N>.0` directory.
+
+- Copy `lts-<N-1>.0/Dockerfile` to `lts-<N>.0/Dockerfile`.
+
+- Check the `FROM` statement, make sure the Ubuntu version matches the Ubuntu version used in the [Stackage Dockerfile](https://github.com/commercialhaskell/stackage/blob/master/Dockerfile).
+
+- Update `GHC_VERSION` to match the version used by the [latest nightly snapshot](https://www.stackage.org/nightly).
+
+- Set `LTS_SLUG` to the [latest nightly snapshot](https://www.stackage.org/nightly) (this will be temporary until the major LTS snapshot is actually released, at which point it will be updated to `lts-<N>.0`).
+
+- Update `PID1_VERSION` and `STACK_VERSION` to the latest versions of those tools.
+
+- Make sure `CUDA_VERSION` and `JVM_PATH` match what [debian-bootstrap.sh](https://github.com/commercialhaskell/stackage/blob/master/debian-bootstrap.sh) uses.
+
+- Update `LLVM_PATH` to the version required for the GHC version.  This will be shown on the download page for the GHC version, which you can reach from https://www.haskell.org/ghc/.  It should match the base directory used in `CLANG_PURE_LLVM_INCLUDE_DIR` in [debian-bootstrap.sh](https://github.com/commercialhaskell/stackage/blob/master/debian-bootstrap.sh) (leaving off the `/include` suffix).
+
+- Update `BOOTSTRAP_COMMIT` to the git commit ID of the latest [debian-bootstrap.sh](https://github.com/commercialhaskell/stackage/blob/master/debian-bootstrap.sh).
+
+- Check for any other `lts-<N>.*/Dockerfile`s and make sure `lts-<N>.0/Dockerfile` includes anything that was updated in those, if they're still relevant for LTS-15 (note that a newer [debian-bootstrap.sh](https://github.com/commercialhaskell/stackage/blob/master/debian-bootstrap.sh) may already include those changes, so check there first).
+
+
+### Perform basic tests
+
+- Build the image: `docker build -t local/stack-build lts-<N>.0/`.
+
+- Ensure that all the directories listed in `PATH`, `CUDA_PATH`, and `CPATH` and any other path-like environment variables actually exist in the image.
+
+- Try building a test package with the new image: `(stack --resolver=nightly new image-test && cd image-test && stack --docker --docker-image=local/stack-build build)`. This should build without needing to install GHC.
+
+- Build the "small" variant: `docker build -t local/stack-build-small --build-arg "VARIANT=small" lts-<N>.0/`.
+
+- Try building a test package with the new small image: `(stack --resolver=nightly new small-image-test && cd small-image-test && stack --docker --docker-image=local/stack-build-small build)`. This should build without needing to install GHC.
+
+### Build real image once major LTS snapshot has been released
+
+- Update `LTS_SLUG` to `lts-<N>.0`
+
+- Update `BOOTSTRAP_COMMIT` to the git commit ID of the latest [debian-bootstrap.sh](https://github.com/commercialhaskell/stackage/blob/master/debian-bootstrap.sh).
+
+- Repeat the tests above, except use `lts-<N>.0` instead of `nightly`.
+
+- Build and push the real images (both standard and `small` variants) using the [build.sh](https://github.com/commercialhaskell/stackage/tree/master/automated/dockerfiles/build.sh) script: `./build.sh --push lts-<N>.0 && ./build.sh --push --small lts-<N>.0`
+
+- Commit and push the new Dockerfile to the `stable` branch.

doc/maintainers/ghc.md

@@ -11,9 +11,14 @@
         git push origin ghc-X.Y.Z-release
 
   * [Publish a new Github release](https://github.com/commercialhaskell/ghc/releases/new)
-    with tag `ghc-X.Y.Z-release` and same name.
+    with tag `ghc-X.Y.Z-release` and same name, with description noting where the binidsts are mirrored from.  E.g.
 
-  * Download all the relevant GHC bindists from https://www.haskell.org/ghc/download_ghc_X_Y_Z and upload them to the just-created Github release (see
+        Unless otherwise indicated, bindists are mirrored from https://downloads.haskell.org/~ghc/
+        * FreeBSD bindists are mirrored from http://distcache.FreeBSD.org/local-distfiles/arrowd/stack-bindists
+        * musl bindists are mirrored from https://github.com/redneb/ghc-alt-libc/releases
+
+
+  * Download all the relevant GHC bindists from their sources, and upload them to the just-created Github release (see
     [stack-setup-2.yaml](https://github.com/fpco/stackage-content/blob/master/stack/stack-setup-2.yaml)
     for the ones we used in the last GHC release).
 
@@ -27,6 +32,16 @@
     and add the new bindists, pointing to the Github release version. Be sure to
     update the `content-length` and `sha1` values.
 
+    Before committing, test using a command like:
+
+        stack --resolver=ghc-X.Y.Z setup --setup-info-yaml=path/to/stackage-content/stack/stack-setup-2.yaml
+
+  * In [stackage-content](https://github.com/fpco/stackage-content), run
+
+        cd stack && ./update-global-hints.hs ghc-X.Y.Z
+
+    and command the changes.
+
 
 ## Building GHC
 

doc/maintainers/releases.md

@@ -14,6 +14,7 @@
 * Look through https://fpcomplete.slack.com/files/U9U8HDGUC/FCM7UN5NJ/notes_on_doc_maintainers_releases_md.txt for hints on how to make this document more clear.
 * If `store` is no longer a dependency, likely can remove from stackage build constraints' `expected-test-failures`
 * Try using Alpine and https://github.com/redneb/ghc-alt-libc/releases to build linux 32-bit static binaries.  This will require upgrading GHC version used (e.g. to 8.6.5).  If this works out, switch over to the same for 64-bit static binaries (rather than static-haskell-nix).
+* Fix the reference to "latest nightly stackage snapshot" below to work with a new workflow based on the GHC-named stack.yaml files.
 
 ## Iterating on release process
 
@@ -119,7 +120,7 @@ Examples:
         * Search for old Stack version, unstable stack version, and the next
           "obvious" possible versions in sequence, and
           `UNRELEASED` and replace with next release version (`X.Y.1`, where Y is odd). 
-          * Do **NOT** update the Dockerfiles in `etc/dockerfiles/stack-build` yet; that will come later)
+          * Do **NOT** update the Dockerfiles in [stackage/automated/dockerfiles](https://github.com/commercialhaskell/stackage/tree/master/automated/dockerfiles/) yet; that will come later)
             * Do __NOT__ update templates in `.github` to point at the new release version yet!
         * Search for old resolvers, set to latest resolver (e.g. in `doc/GUIDE.md` where it references the "currently the latest LTS")
         * Look for any links to "latest" (`latest/`) documentation, replace with version tag
@@ -246,17 +247,17 @@ for requirements to perform the release, and more details about the tool.
 
     - Update fpco/stack-build Docker images with new version
 
-      * Add `etc/dockerfiles/stack-build/lts-X.Y/Dockerfile` (where `X.Y` is the latest stackage LTS version), containing (note where X.Z is the previous LTS version, and X.Y.Z is the newly released stack version)
+      * Under [stackage/automated/dockerfiles](https://github.com/commercialhaskell/stackage/tree/master/automated/dockerfiles/), add `lts-X.Y/Dockerfile` (where `X.Y` is the latest stackage LTS version), containing (note where X.Z is the previous LTS version, and X.Y.Z is the newly released stack version)
 
         ```
         FROM fpco/stack-build:lts-X.Z
         ARG STACK_VERSION=X.Y.Z
         RUN wget -qO- https://github.com/commercialhaskell/stack/releases/download/v$STACK_VERSION/stack-$STACK_VERSION-linux-x86_64.tar.gz | tar xz --wildcards --strip-components=1 -C /usr/local/bin '*/stack'
         ```
 
-      * Run `./build.sh lts` and test that the new image has the new version of Stack.
+      * Run `./build.sh lts-X.Y` and test that the new image has the new version of Stack.
 
-      * Run `./build.sh --push lts && ./build.sh --push --small lts` to push the new image to the registry.
+      * Run `./build.sh --push lts-X.Y && ./build.sh --push --small lts-X.Y` to push the new image to the registry.
 
 
 * Delete the RC branch (locally and on origin).  E.g. `git branch -d rc/vX.Y; git push origin :rc/vX.Y`.

doc/nix_integration.md

@@ -86,12 +86,11 @@ You can list all known Haskell compilers in Nix with the following:
 $ nix-instantiate --eval -E "with import <nixpkgs> {}; lib.attrNames haskell.compiler"
 ```
 
-Alternatively, install `nix-repl`, a convenient tool to explore
+Alternatively, use `nix repl`, a convenient tool to explore
 nixpkgs:
 
 ```sh
-$ nix-env -i nix-repl
-$ nix-repl
+$ nix repl
 ```
 
 In the REPL, load nixpkgs and get the same information through
@@ -102,7 +101,7 @@ nix-repl> :l <nixpkgs>
 nix-repl> haskell.compiler.ghc<Tab>
 ```
 
-You can type and evaluate any nix expression in the nix-repl, such as
+You can type and evaluate any nix expression in the nix repl, such as
 the one we gave to `nix-instantiate` earlier.
 
 **Note:** currently, stack only discovers dynamic and static libraries

doc/travis-complex.yml

@@ -8,8 +8,8 @@
 # Copy these contents into the root directory of your Github project in a file
 # named .travis.yml
 
-# Use new container infrastructure to enable caching
-sudo: false
+# Run jobs on Linux unless "os" is specified explicitly.
+os: linux
 
 # Do not choose a language; we provide our own build tools.
 language: generic
@@ -30,9 +30,9 @@ cache:
 # cache file per set of arguments.
 #
 # If you need to have different apt packages for each combination in the
-# matrix, you can use a line such as:
+# job matrix, you can use a line such as:
 #     addons: {apt: {packages: [libfcgi-dev,libgmp-dev]}}
-matrix:
+jobs:
   include:
   # We grab the appropriate GHC and cabal-install versions from hvr's PPA. See:
   # https://github.com/hvr/multi-ghc-travis