Rebuild project when dependencies have changed, even if they are the same version?

[DanBurton]

Example:

I built a project with stack build. Then I added the following (which are in the snapshot) to my extra-deps:

  - data-default-class-0.0.1
  - cookie-0.4.1.5
  - data-default-0.5.3
  - data-default-instances-base-0.0.1
  - data-default-instances-containers-0.0.1
  - data-default-instances-dlist-0.0.1
  - data-default-instances-old-locale-0.0.1
  - wai-extra-3.0.7.1
  - yesod-core-1.4.9.1

Then I ran stack build again. It installed the extra-deps, as expected, but did not rebuild the project.

Expected: it should have rebuilt the project?

(See #120 for superfluous context.)

[snoyberg]

I think so, pinging @chrisdone.

[chrisdone]

I'm not sure what to do here. Perhaps invalidating all local user targets which depend on local dep targets that are about to be installed?

[snoyberg]

Continuing with what we've been doing until now... it seems like we should have each package depend on a specific package ID for its dependencies, and if the build plan includes a different package ID, it should trigger a rebuild. I don't know the Shake side well enough to express this with the right terminology, however.

[chrisdone]

In this exact use-case the package id doesn't change because the package content is the same.

Snapshot db:

chris@retina:~/Work/stack$  ghc-pkg -f ~/.stack/snapshots/x86_64-linux/lts-2.9/7.8.4/pkgdb/ describe http-client-tls-0.2.2 | grep id:
id: http-client-tls-0.2.2-c47fc9660b746f22f229a66fe205c8e4

Local db:

chris@retina:~/Work/stack$ ghc-pkg -f .stack-work/install/x86_64-linux/lts-2.9/7.8.4/pkgdb/ describe http-client-tls-0.2.2 | grep id:
id: http-client-tls-0.2.2-c47fc9660b746f22f229a66fe205c8e4 

So installing a local package wouldn't be considered a change. It also produces the same result over subsequent reinstalls:

chris@retina:~/Work/stack$ ghc-pkg -f .stack-work/install/x86_64-linux/lts-2.9/7.8.4/pkgdb/ describe path | grep id:
id: path-0.4.0-1d297391952fa79c9441a73815b03e3c
chris@retina:~/Work/stack$ stack build # switch to path-0.5.0, removes path-0.4.0
path-0.4.0: unregistering
stack-0.0.0: unregistering
…
chris@retina:~/Work/stack$ stack build # enable path-0.4.0 again
path-0.5.0: unregistering
stack-0.0.0: unregistering
…
chris@retina:~/Work/stack$ ghc-pkg -f .stack-work/install/x86_64-linux/lts-2.9/7.8.4/pkgdb/ describe path | grep id:
id: path-0.4.0-1d297391952fa79c9441a73815b03e3c
chris@retina:~/Work/stack$ 

[chrisdone]

If the versions change, depending on package-ver should be enough, though. So the targets would always be name-version, I think that would at least handle version changes. In here: needDependencies.

[chrisdone]

Actually the targets are already versioned:

chris@retina:~/Work/stack$ ls .stack-work/unpacked/
cookie-0.4.1.5         http-conduit-2.1.5    optparse-simple-0.0.3  path-0.5.0
http-client-tls-0.2.2  monad-unlift-0.1.1.0  path-0.4.0

Let me confirm this actually works if I change path from 0.4.0 to 0.5.0 WITHOUT the unregistering code enabled.

[chrisdone]

It seems that Shake will not re-run the dependency calculation. Introducing new dependencies doesn't seem to shake dependent targets into life.

chris@retina:~/Work/stack$ stack build
path-0.4.0: unregistering [no-op!]
Make plan for monad-unlift [Wanted]
Make plan for optparse-simple [Wanted]
Make plan for path [Wanted]
Make plan for stack [Wanted]
Plan is at least evaluated...

# Change path version to 0.4.0 

chris@retina:~/Work/stack$ stack build
path-0.5.0: unregistering [no-op!]
Make plan for monad-unlift [Wanted]
Make plan for optparse-simple [Wanted]
Make plan for path [Wanted]
Make plan for stack [Wanted]
Plan is at least evaluated...
buildTarget action for path is evaluated...
configureTarget action for path is evaluated...
path-0.4.0: configure
path-0.4.0: build
path-0.4.0: install

# Remove the path-0.5.0 gen file
chris@retina:~/Work/stack$ rm .stack-work/unpacked/path-0.5.0/dist-stack/Cabal-1.18.1.5/stack.gen

chris@retina:~/Work/stack$ stack build
path-0.5.0: unregistering [no-op!]
Make plan for monad-unlift [Wanted]
Make plan for optparse-simple [Wanted]
Make plan for path [Wanted]
Make plan for stack [Wanted]
Plan is at least evaluated...
buildTarget action for stack is evaluated...  # stack's dependency list is actually evaluated now
stack depends on monad-unlift-0.1.1.0
stack depends on optparse-simple-0.0.3
stack depends on path-0.4.0
configureTarget action for stack is evaluated...
stack depends on monad-unlift-0.1.1.0
stack depends on optparse-simple-0.0.3
stack depends on path-0.4.0
stack-0.0.0: configure
stack-0.0.0: build
Building stack-0.0.0...
Preprocessing library stack-0.0.0...
In-place registering stack-0.0.0...
Preprocessing executable 'stack' for stack-0.0.0...
Linking dist-stack/Cabal-1.18.1.5/build/stack/stack ...
stack-0.0.0: install
chris@retina:~/Work/stack$ 

I guess I need to go back to the Shake drawing board and find out if there's a way to always evaluate the set of dependencies.

[chrisdone]

Okay, through some surgery and use of an oracle I've managed to get Shake to pay attention to dependency changes:

chris@retina:~/Work/stack$ stack build
path-0.4.0: unregistering [no-op!]
Make plan for monad-unlift [Wanted]
Make plan for optparse-simple [Wanted]
Make plan for path [Wanted]
Make plan for stack [Wanted]
Plan is at least evaluated...
configureTarget action for stack is evaluated...
buildTarget action for path is evaluated...
stack depends on monad-unlift-0.1.1.0
configureTarget action for path is evaluated...
stack depends on optparse-simple-0.0.3
stack depends on path-0.5.0
path-0.5.0: configure
path-0.5.0: build
path-0.5.0: install
stack-0.0.0: configure
buildTarget action for stack is evaluated...
stack-0.0.0: build
Building stack-0.0.0...
Preprocessing library stack-0.0.0...
In-place registering stack-0.0.0...
Preprocessing executable 'stack' for stack-0.0.0...
[4 of 4] Compiling Main             ( src/main/Main.hs, dist-stack/Cabal-1.18.1.5/build/stack/stack-tmp/Main.o ) [/home/chris/Work/stack/.git/index changed]
Linking dist-stack/Cabal-1.18.1.5/build/stack/stack ...
stack-0.0.0: install
chris@retina:~/Work/stack$ stack build
path-0.4.0: unregistering [no-op!]
Make plan for monad-unlift [Wanted]
Make plan for optparse-simple [Wanted]
Make plan for path [Wanted]
Make plan for stack [Wanted]
Plan is at least evaluated...
chris@retina:~/Work/stack$ 

[chrisdone]

Pushed a commit that rebuilds with the versions change. I think the only way to rebuild in this scenario would be to include some kind of hash of the package database path as part of the shake targets. Is it worth it?

[chrisdone]

Actually let me test this scenario out. This change might've fixed it.

[chrisdone]

Yeah, this seems to work:

chris@retina:~/Work/stack$ stack build
Installing dependencies: appar-0.1.4, byteorder-1.0.4, clientsession-0.9.1.1, cprng-aes-0.6.1, crypto-api-0.13.2, easy-file-0.2.1, entropy-0.3.6, http-date-0.0.6.1, iproute-1.3.2, setenv-0.1.1.3, shakespeare-2.0.5, simple-sendfile-0.2.18, skein-1.0.9.3, stringsearch-0.3.6.6, unix-compat-0.4.1.4, unix-time-0.3.5, vault-0.3.0.4, wai-3.0.2.3, wai-logger-2.2.4, warp-3.0.13, word8-0.1.2
Downloading appar-0.1.4
Downloading byteorder-1.0.4
Downloading clientsession-0.9.1.1
Downloading cprng-aes-0.6.1
Downloading crypto-api-0.13.2
Downloading easy-file-0.2.1
Downloading entropy-0.3.6
Downloading http-date-0.0.6.1
Downloading iproute-1.3.2
Downloading setenv-0.1.1.3
Downloading shakespeare-2.0.5
Downloading simple-sendfile-0.2.18
Downloading skein-1.0.9.3
Downloading stringsearch-0.3.6.6
Downloading unix-compat-0.4.1.4
Downloading unix-time-0.3.5
Downloading vault-0.3.0.4
Downloading wai-3.0.2.3
Downloading wai-logger-2.2.4
Downloading warp-3.0.13
Downloading word8-0.1.2
All dependencies unpacked
http-date-0.0.6.1: configure
appar-0.1.4: configure
http-date-0.0.6.1: build
…
stack-0.0.0: configure
stack-0.0.0: build
Building stack-0.0.0...
Preprocessing library stack-0.0.0...
In-place registering stack-0.0.0...
Preprocessing executable 'stack' for stack-0.0.0...
[4 of 4] Compiling Main             ( src/main/Main.hs, dist-stack/Cabal-1.18.1.5/build/stack/stack-tmp/Main.o ) [/home/chris/Work/stack/.git/index changed]
Linking dist-stack/Cabal-1.18.1.5/build/stack/stack ...
stack-0.0.0: install

@DanBurton Confirm?