Shake tries to build existing target

[cblp]

{-# LANGUAGE DeriveDataTypeable #-}

import Control.Exception  ( Exception, throwIO )
import Data.Typeable      ( Typeable )
import Development.Shake  ( (%>), liftIO, shakeArgs, shakeOptions, want )
import System.Directory   ( doesFileExist )

main = shakeArgs shakeOptions $ do
    want ["a"]
    "a" %> \_ -> do
        liftIO $ create "a"

data FileExists = FileExists FilePath
    deriving (Show, Typeable)
instance Exception FileExists

create :: FilePath -> IO ()
create name = do
    exists <- doesFileExist name
    if exists then
        throwIO $ FileExists name
    else
        writeFile name ""

$ ./build.hs
Error when running Shake build system:
* a
FileExists "a"

How can I tell Shake that the file already exists and it doesn't need to reset it?

[ndmitchell]

The above pattern is a bit weird. Is it the case that a is always created exactly the same? If so, I'd do:

import System.Directory as IO

create name = do
    exists <- IO.doesFileExist name
    unless exists $ writeFile name ""

Or maybe there's something else underlying that you are trying to express? I think some more context would probably be useful.

[cblp]

It may be any other action, for instance, cabal:

$ cabal get shake
cabal: The directory "shake-0.15/" already exists, not unpacking.
$ echo $?
1

import Development.Shake

main = shakeArgs shakeOptions $ do
    let pkgName = "shake"
        pkgVersion = "0.15"
        pkgId = pkgName ++ "-" ++ pkgVersion
    want [pkgId]
    pkgId %> \_ ->
        cmd "cabal" "get" pkgId

$ ./build.hs
# cabal (for shake)
cabal: The directory "shake-0.15/" already exists, not unpacking.
Error when running Shake build system:
* shake
user error (Development.Shake.cmd, system command failed
Command: cabal get shake
Exit code: 1
Stderr:
cabal: The directory "shake-0.15/" already exists, not unpacking.
)

[ndmitchell]

So I guess you want:

 b <- IO.doesDirectoryExist pkgId
 unless b $ cmd "cabal" "get" pkgId

The "weird bit" I was referring to was the exceptions, but it looks like you were just using those to model Cabal, so aren't part of what you were intending to do in Shake.

[cblp]

Yes, I already use this workaround.

But what about this bug? Isn't Shake the tool to do one particular thing — to skip work if some objects are already made? If I have to call doesDirectoryExist before any target, so what Shake will do?

[ndmitchell]

This is the expected behaviour. Shake skips targets it has already made, not that already exist. Since Shake wasn't responsible for creating that file, it reruns the rule. This is in contrast to Make which has no idea whether it made the file or not, so always skips files if it finds them.

One reason Shake has to rerun such targets is so that it can track what dependencies they have, so it knows when to rerun them in future. More generally, it lets you switch from a file being user created to generated without accidentally sticking with the user created file, and thinking the generator is working when it's not.

[cblp]

I see. Shake cannot detect dependencies without running the recipe.

Is there any way to tell Shake that this particular target mustn't be rebuilt if exists? Without doesDirectoryExist, because it is strange. Maybe something like

   pkgId %> \_ -> do
        trustFilesystem
        cmd "cabal" "get" pkgId

?

[cblp]

As for me, it is not expected behaviour when I read "Shake is an alternative to make".

[cblp]

Of course, mentioned above hypothetic trustFilesystem should not only skip the recipe, but mark this target as built by Shake.

Consider it a feature proposal.

[ndmitchell]

You could define a rule that skipped if the file already existed, something like:

(???>) pat act = pat %> \out ->
    b <- doesFileExist out
    unless b $ act out

But I think it's unlikely to be common, since you usually want to actually have built the rule with Shake. I view Shake as an alternative to Make in the sense you can build stuff with it, but have explicitly tried to fix the flaws with Make. Since Shake is parameterisable over rule types you can certainly define such a rule (or define a combinator), but I think it has to boil down to doesFileExist in the end.

[cblp]

But I think it's unlikely to be common

I beg to differ. In my practice, I've never meet cases when we need to rebuild an object when human knows it is up-to-date but builder doesn't. Moreover, there are many cases when rebuilding an object is undesirable even if some metadata are accidentally lost (e. g. when building takes hours or days).

The problem of make is that it has very limited ways to deal with metadata. — It is impossible or very difficult to change rules or dependecies in the run time.

The problem of Shake that it mixes metadata gathering into the build rules. — It causes unnecessary actions spending extra resources.

Solution: what about dividing build rules into two steps: (a) gathering metadata and (b) building an object? Users must be warned that metadata step may be called more often than they expect, but there must be a guarantee that build rules will not be called when it is not really needed.

Should we discuss it in https://groups.google.com/forum/#!forum/shake-build-system?

[ndmitchell]

Sounds reasonable to discuss with others on the mailing list. Until now Shake has very much been "in control" of what happens, it sounds like you're interested in the user being more in control and Shake taking its cues from the user - a different focus and interesting to see what people think.

[Mathnerd314]

Running shake --touch will import a into the database without trying to create it. I think that solves the issue of "Tell Shake that the file already exists and it doesn't need to rebuild it."