infer --package options from imports in standalone scripts

[simonmichael]

In the stack script excerpt below, I started with just a few --package options. I'm using it in a non-cabal project, so there's no local .stack-work directory and it's using my global stack package db, which often has the required packages already. So imports tend to work even if I don't specify all the required packages, which is not good as it means the script may not run on another machine. (Possibly worth a separate feature request: allow imports only from explicitly specified packages, as cabal does).

So I started using PackageImports and writing the package explicitly in the imports, to find which additional --package declarations were needed (many). This also seems useful simply as documentation, it's often unclear which package a module comes from. Now there is obvious redundancy, and I'm wondering if we could omit the --package declarations and have stack infer them from the packages mentioned in imports (or even just from the imported module names with no package names needed).

#!/usr/bin/env stack   # https://www.haskell.org/downloads#stack
{- stack runghc
   --package base-prelude
   --package bytestring
   --package directory
   --package docopt
   --package extra
   --package filepath
   --package http-client
   --package http-client-tls
   --package lens
   --package safe
   --package time
   --package netrc
   --package process
   --package wreq
-}
{-
Usage: see below
-}

{-# LANGUAGE PackageImports #-}

import                              Prelude ()
import           "base-prelude"     BasePrelude
import           "base"             System.IO
import qualified "bytestring"       Data.ByteString.Char8 as B8
import qualified "bytestring"       Data.ByteString.Lazy.Char8 as LB8
import           "directory"        System.Directory
import           "docopt"           System.Console.Docopt
import           "extra"            Data.List.Extra (takeEnd)
import           "filepath"         System.FilePath
import           "http-client"      Network.HTTP.Client (managerResponseTimeout, createCookieJar, HttpException(..))
import           "http-client-tls"  Network.HTTP.Client.TLS (tlsManagerSettings)
import           "lens"             Control.Lens hiding (argument)
import           "netrc"            Network.NetRc
import           "process"          System.Process
import           "safe"             Safe (readMay)
import           "time"             Data.Time
import           "wreq"             Network.Wreq
import qualified "wreq"             Network.Wreq.Session as S
import           "wreq"             Network.Wreq.Types (Postable)

[harendra-kumar]

This is something that I would really like to have. From an implementation perspective it will require a bit more parsing now. Till now we have had simple attoparsec parser for the hashbang line. For this job we may need a real source parser to handle all the cases correctly.

[mgsloan]

This would also be useful in general for determining package dependencies. There are a few ways to achieve this, preferred options earlier in the list:

1. Add a plugin to haskell-ide-engine which dumps the list of imports. I'd prefer to use info from a properly configured GHC, after preprocessing has been done, etc.

2. Parse GHC error messages. I think it gives all missing module errors in one go, at least in recent GHC. This should be tested before going with this implementation approach.

3. Use a haskell-src-exts parser which I added, ModuleHeadAndImports. While this would probably work well for these single file scripts, I'd really like to have something that works for complicated projects that use features like preprocessing.

The main issue with (1) is that HIE would need to be made to work reliably for all stack projects, and I have had some difficulty with that in the past. This is something I want to have happen for many other reasons, though. This would also mean that if HIE isn't built yet, it'll need to be built before autodetecting dependencies.

The main issue with (2) is that error message format changes between GHC versions. Also, I'm not sure it will give the errors for all imports transitively in the project. It would suck to have to do iterations with the compiler.

(3) has a number of issues. We'd need to run CPP somehow or try to munge it out (potentially including imports that aren't actually used). HSE is a large dependency which gives people trouble compiling it (OOM issues). It wouldn't handle. For this single file script usecase, it'd be the easiest way to get this implemented. But if we want to go any further with this, it'll start getting hacky and complicated.

Possibly worth a separate feature request: allow imports only from explicitly specified packages, as cabal does

Yep, this would be good. It's tracked by #1208

[snoyberg]

Does the new script command address the goals you hide in mind here?

[simonmichael]

I think script solves the first issue (restrict imports to explicitly declared packages only) and not the second (redundant package lists in stack command line and imports).

[snoyberg]

The script command parses imports to determine which packages should be available, so I believe it does solve the second.

[simonmichael]

Oh, great! I just tested a little and have some quick notes: - with one or more --package options, stack script does restrict imports to the declared packages. - with no --package options, it detected and installed the right packages just based on the import statements. The PackageImports-style package declarations weren't needed either. - it showed the "No packages provided, using experimental import parser" message at default verbosity. Maybe we'd rather see this only at --verbosity=info and up.

[snoyberg]

I can't reproduce your first point, e.g.:

#!/usr/bin/env stack
-- stack script --resolver lts-8.12 --package base
import Control.Monad.Reader

main = return ()

results in:

foo.hs:3:1: error:
    Failed to load interface for ‘Control.Monad.Reader’
    It is a member of the hidden package ‘mtl-2.2.1’.
    Use -v to see a list of the files searched for.

Your third point is already implemented on master.

[simonmichael]

That's what I'm saying, it works. All looks good! Off topic, but while I'm testing, I still find myself wishing - that I didn't have to specify a resolver sometimes. Eg at times when it seems any newer snapshot would work equally well, and by specifying one I feel I'm forcing users to do a lot of unnecessary downloading and recompiling. - that I could specify specific versions of packages with the script command (as I can with runghc), to force stack to use just-released versions that aren't in stackage lts (or nightly) yet. These seem to come up most when I'm writing install instructions for newly released software.

[simonmichael]

PS and maybe these are just incompatible with repeatability.. I'm thinking aloud in case there might still be a few more conveniences it could allow without sacrificing that.

[snoyberg]

Ah, I misread your comment.

The two other comments you gave are, in fact, specifically avoided due to concerns around reproducibility. With the upcoming extensible snapshots feature, however, it should be possible to define a custom snapshot with the just-released (or even not-yet-released) versions of your packages and include those in a script.

[simonmichael]

Cool beans. This new auto-detection of required packages seems a pretty great feature, by the way. One less thing a haskell coder has to think about. Thanks!

[snoyberg]

Cool, thanks for the feedback. You OK with closing out this issue then?

[simonmichael]

Absolutely.