New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Haskell source pre-processing in Haskell no longer works #1541

Closed
ygale opened this Issue Oct 13, 2013 · 6 comments

Comments

Projects
None yet
6 participants
@ygale
Copy link
Contributor

ygale commented Oct 13, 2013

The standard technique for pre-processing Haskell source code using Haskell has stopped working in recent versions of Cabal. The technique is as follows:

The cabal file is structured like this:

Name: foo
...
Executable foo-meta
  ...
Library
  ...
Executable foo
  ...

Then you pass "-F -pgmF dist/build/foo/foo-meta" to GHC for the source file you want to pre-process, e.g., using an OPTIONS_GHC pragma.

The effect is that first cabal builds foo-meta, then uses it to build the library, then builds the final executable.

But now, Cabal refuses to build the artifacts in the order I specified in the Cabal file. It always tries to build the library first, before the foo-meta executable, and the build fails.

Note: We have only tried this so far inside a sandbox. However, we have verified that the problem is definitely that cabal is not even trying to build foo-meta, not a problem with differences in the path to the foo-meta executable in a sandbox.

@23Skidoo

This comment has been minimized.

Copy link
Member

23Skidoo commented Oct 13, 2013

Can't you just release foo-meta as a separate package?

@ygale

This comment has been minimized.

Copy link
Contributor

ygale commented Nov 12, 2013

That's really ugly, for a few reasons.

  1. The preprocessing code and the main code are all one code base. Maintaining them as two separate packages is counterintuitive.
  2. You need to coordinate the version numbers of the two packages manually to ensure that the preprocessor gets rebuilt, or not, as needed.
  3. There is an awkward extra requirement to set up some external relationship between the two packages. For example, you can create sandbox add-source links manually every time you check out the combined project. Or you can give up on using a single cabal install command and write a build script that builds things in the right order. Or upload (manually) the packages frequently to hackage or a local yackage server so that the preprocessor will build. But then what happens if there are code dependencies and you make a breaking change that affects both the preprocessor and the main code at the same time? You can't compile the main code until you have compiled the preprocessor, but you can't compile the preprocessor until you have uploaded the new version of the main code.

This being Haskell, I agree that it would be nice to move away from the imperative feel of requiring the artifacts to built in the order they physically appear in the cabal file. Perhaps we should add some cabal file syntax for expressing a dependency of one artifact on another defined in the same cabal file. In one direction we already have something primitive: an executable can list its own package name as a build dependency, causing the library to be built before this executable.

But in the meantime, cabal has broken behavior that was previously working.

@bermanjosh

This comment has been minimized.

Copy link

bermanjosh commented Nov 12, 2013

I may have found a bug which is related to why cabal seems to be ignoring the build order.

in: Cabal/Distribution/Simple/LocalBuildInfo.hs, lines 342 - 357:

{-# DEPRECATED withComponentsLBI "Use withAllComponentsInBuildOrder" #-}
withComponentsLBI :: PackageDescription -> LocalBuildInfo
                  -> (Component -> ComponentLocalBuildInfo -> IO ())
                  -> IO ()
withComponentsLBI = withAllComponentsInBuildOrder

-- | Perform the action on each buildable 'Library' or 'Executable' (Component)
-- in the PackageDescription, subject to the build order specified by the
-- 'compBuildOrder' field of the given 'LocalBuildInfo'
withAllComponentsInBuildOrder :: PackageDescription -> LocalBuildInfo
                              -> (Component -> ComponentLocalBuildInfo -> IO ())
                              -> IO ()
withAllComponentsInBuildOrder pkg lbi f =
    sequence_
      [ f (getComponent pkg cname) clbi
      | (cname, clbi) <- allComponentsInBuildOrder lbi ]

Based on the comments of the withAllComponentsInBuildOrder function, it is looking for a field compBuildOrder of the LocalBuildInfo type, which seems to be defined on lines 116-160. However, I don't see a compBuildOrder field in that definition.

data LocalBuildInfo = LocalBuildInfo {
        configFlags   :: ConfigFlags,
        -- ^ Options passed to the configuration step.
        -- Needed to re-run configuration when .cabal is out of date
        extraConfigArgs     :: [String],
        -- ^ Extra args on the command line for the configuration step.
        -- Needed to re-run configuration when .cabal is out of date
        installDirTemplates :: InstallDirTemplates,
                -- ^ The installation directories for the various differnt
                -- kinds of files
        --TODO: inplaceDirTemplates :: InstallDirs FilePath
        compiler      :: Compiler,
                -- ^ The compiler we're building with
        hostPlatform  :: Platform,
                -- ^ The platform we're building for
        buildDir      :: FilePath,
                -- ^ Where to build the package.
        --TODO: eliminate hugs's scratchDir, use builddir
        scratchDir    :: FilePath,
                -- ^ Where to put the result of the Hugs build.
        componentsConfigs   :: [(ComponentName, ComponentLocalBuildInfo, [ComponentName])],
                -- ^ All the components to build, ordered by topological sort, and with their dependencies
                -- over the intrapackage dependency graph
        installedPkgs :: PackageIndex,
                -- ^ All the info about the installed packages that the
                -- current package depends on (directly or indirectly).
        pkgDescrFile  :: Maybe FilePath,
                -- ^ the filename containing the .cabal file, if available
        localPkgDescr :: PackageDescription,
                -- ^ The resolved package description, that does not contain
                -- any conditionals.
        withPrograms  :: ProgramConfiguration, -- ^Location and args for all programs
        withPackageDB :: PackageDBStack,  -- ^What package database to use, global\/user
        withVanillaLib:: Bool,  -- ^Whether to build normal libs.
        withProfLib   :: Bool,  -- ^Whether to build profiling versions of libs.
        withSharedLib :: Bool,  -- ^Whether to build shared versions of libs.
        withDynExe    :: Bool,  -- ^Whether to link executables dynamically
        withProfExe   :: Bool,  -- ^Whether to build executables for profiling.
        withOptimization :: OptimisationLevel, -- ^Whether to build with optimization (if available).
        withGHCiLib   :: Bool,  -- ^Whether to build libs suitable for use with GHCi.
        splitObjs     :: Bool,  -- ^Use -split-objs with GHC, if available
        stripExes     :: Bool,  -- ^Whether to strip executables during install
        progPrefix    :: PathTemplate, -- ^Prefix to be prepended to installed executables
        progSuffix    :: PathTemplate -- ^Suffix to be appended to installed executables
  } deriving (Read, Show)

Sorry if I missed something; I'm still relatively new to Haskell.
Thanks

@dcoutts

This comment has been minimized.

Copy link
Member

dcoutts commented Dec 9, 2013

@ygale we already have a system for this, which is the build-tool field. You can use that to list that your lib or exe depends on the build tool defined within the same package.

Now having said that, while I wrote that code I've never actually tested it (or if I did, I don't remember!) So could you give that a go and tell us if it works, and we can close this ticket, or re-purpose it.

And about having broken existing things, we never promised the components would be built in order, and using -F -pgmF dist/build/foo/foo-meta is not actually supported (because we do not promise that the dist dir is called dist, nor the internal layout of files within that dir). This build-tools thing has been there for a while however, so it's safe to use that (assuming it is actually working). The "right way" to do pre-processors within the same package is to use code in Setup.hs to declare the pre-processor.

@23Skidoo

This comment has been minimized.

Copy link
Member

23Skidoo commented Dec 12, 2013

@dcoutts Yes, this seems to work: https://gist.github.com/23Skidoo/7930870

Leaving the issue open for now because I want to add a test case for this.

@ttuegel ttuegel added the type: bug label Apr 23, 2015

@ttuegel ttuegel added this to the Cabal-1.24 milestone Apr 23, 2015

@23Skidoo 23Skidoo modified the milestones: Cabal 1.24, Cabal 1.26 Feb 21, 2016

@ezyang

This comment has been minimized.

Copy link
Contributor

ezyang commented Mar 31, 2016

This is a little known feature of Cabal, not mentioned in the manual, and rarely known by many who have this problem in the wild.

Additionally, it would be quite nice if we could support the preprocessor use-case without needing people to add a custom Setup.hs. Perhaps we can support -pgmF style: what if Cabal adds every mentioned build-tool to the PATH?

ezyang added a commit to ezyang/cabal that referenced this issue Mar 31, 2016

Fix build-tools ordering regression (haskell#3257, haskell#1541)
When converting the component graph to operate in terms of UnitIds
instead of CNames I accidentally introduced a regression where we
stopped respecting build-tools when determining an ordering to
build things.  This commit fixes the regression (though perhaps
not in the most clean/performant way you could manage it.)  It
also fixes a latent bug if internal libraries aren't processed
in the correct order.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Mar 31, 2016

Fix build-tools ordering regression (haskell#3257, haskell#1541)
When converting the component graph to operate in terms of UnitIds
instead of CNames I accidentally introduced a regression where we
stopped respecting build-tools when determining an ordering to
build things.  This commit fixes the regression (though perhaps
not in the most clean/performant way you could manage it.)  It
also fixes a latent bug if internal libraries aren't processed
in the correct order.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Mar 31, 2016

Fix build-tools ordering regression (haskell#3257, haskell#1541)
When converting the component graph to operate in terms of UnitIds
instead of CNames I accidentally introduced a regression where we
stopped respecting build-tools when determining an ordering to
build things.  This commit fixes the regression (though perhaps
not in the most clean/performant way you could manage it.)  It
also fixes a latent bug if internal libraries aren't processed
in the correct order.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Mar 31, 2016

Fix build-tools ordering regression (haskell#3257, haskell#1541)
When converting the component graph to operate in terms of UnitIds
instead of CNames I accidentally introduced a regression where we
stopped respecting build-tools when determining an ordering to
build things.  This commit fixes the regression (though perhaps
not in the most clean/performant way you could manage it.)  It
also fixes a latent bug if internal libraries aren't processed
in the correct order.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit that referenced this issue Apr 1, 2016

Fix build-tools ordering regression (#3257, #1541)
When converting the component graph to operate in terms of UnitIds
instead of CNames I accidentally introduced a regression where we
stopped respecting build-tools when determining an ordering to
build things.  This commit fixes the regression (though perhaps
not in the most clean/performant way you could manage it.)  It
also fixes a latent bug if internal libraries aren't processed
in the correct order.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 4, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 5, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 5, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 5, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 7, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 7, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 7, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 11, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 11, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 11, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 12, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 12, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 14, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 15, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 18, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 21, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

ezyang added a commit to ezyang/cabal that referenced this issue Aug 21, 2016

Fix haskell#1541, by adding internal build-tools to PATH.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

@ezyang ezyang closed this in 6764810 Aug 21, 2016

@ezyang ezyang modified the milestones: Cabal 2.0, 2.0 (planned for next feature release) Sep 6, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment