Rework skip/rebuild flags

[ndmitchell]

The current set of flags for specifying what should be rebuilt are confusing because they are inherited from make, and are all described in terms of filestamps. I propose the following flags and semantics:

• --rebuild=pat - in this build, if you encounter a file matching pat, rebuild it even if it didn't otherwise need rebuilding.
• --skip=pat - in this build, if you encounter a file matching pat, do not rebuild it even if it's dependencies have changed. If you run again without --skip the file will rebuild again. If the file is not present it will still be rebuilt.
• --skip-always=pat -- in this build, if you encounter a file matching pat, mark it as fresh (update the stored timestamps and dependencies of versions), so it won't rebuild again in future runs (unless it changes again).

[snowleopard]

One point is unclear. Say, we have a dependency chain A <- B <- C. Note, there is a transitive dependency of C on A, but C needs only B explicitly.

Now I modify A and run Shake with --skip=B.

Will C be rebuilt? On the one hand, it is transitively dirty, but on the other hand we skip its only explicit dependency, and I think it doesn't make sense to rebuild it without rebuilding B first.

Not rebuilding C seems to be the right thing to do in this scenario, and I think we'd like to avoid explicitly telling to skip all dependencies with --skip=B --skip=C. Then --skip=stage1/** will indeed work very nicely for the GHC build system, that is Stage2 targets won't be rebuilt because of transitive dependencies on everything through inplace/bin/ghc-stage1.

[ndmitchell]

We would not rebuild C - C depends on B, B hasn't changed (because we skipped it), and thus C doesn't rebuild.

[snowleopard]

Thanks!

A follow-up question. If we modify the above scenario so that A <- B and [A, B] <- C, that is C now explicitly needs both A and B, then we probably should rebuild C when running with --skip=B?

[ndmitchell]

Correct on the follow up - assuming A changes then C rebuilds because of its dependency on A. But we still don't rebuild B.

[snowleopard]

I see, makes sense. This could probably lead to strange results and possibly failures, because the build action for C will see an inconsistent state of the world (A has changed, but B hasn't), but this is probably an expected risk when using the --skip feature.

[ndmitchell]

Yep, and in fact the whole point of --skip is to see inconsistent results. The nice thing about --skip is a future build without it will put it all back in the right place. Of these, --skip-always is the truly dangerous one.

[ndmitchell]

Merging #307:

Imagine you want to temporarily skip building generated files, so you can manually tweak them. The way to do it should be to use the make mode skip flags, which Shake mirrors from make. Unfortunately, the flags are a bit weird, maybe --what-if does it, but may not - I need to figure out the semantics before I recommend it. Perhaps a better design would be:

 shake --ignore=*.java

That would not rebuild any .java files, even if they are dirty. However, once the flag was no longer passed, shake would fix everything up. Fortunately, that's quite easy in shake. Roughly:

• Add an oracle that has a list of all the ignore flags.
• When you have --ignore add a priority 100 rule that matches the pattern, depends on the oracle, does no IO actions, and simply checks the file is already present.

With that, you get the behaviour we want. Adding the ignore flag to shake proper is probably a good idea.

[ndmitchell]

In response to the text of #307, it seems like I the suggested --ignore is --skip - I think I currently prefer skip as a name. The implementation is a nice way to implement this feature "on the outside", but I'm not sure that's necessary.

[ndmitchell]

Plan is these will be implemented in Rules.hs of #453.

[ndmitchell]

Note that in #427 I changed what AssumeSkip does, which I assume means I broke it (I'm not sure I know enough to say I did or didn't break it, but it's a reasonable working assumption).

[ndmitchell]

I figured out where the equivalent piece would be in the new world, so I fixed up AssumeSkip. That now makes the difference between AssumeSkip and AssumeClean much clearer - roughly AssumeSkip == skip and AssumeClean == skip-always.

[michalt]

I've started talking with @snowleopard about changing Hadrian to add something like make 1 and make 2 that's possible in the current GHC build system (i.e., only rebuild stage 1 or stage 2 compiler).

So we were wondering what is the current status of this issue? Also, are there any workarounds that we could use in the meantime? Thanks!

[ndmitchell]

I'm actively hacking as we speak! How long do I have before you need workarounds?

[snowleopard]

Well, we are synchronised at ZuriHac right now, which will last for two more days. It would be great to implement this new feature while @michalt and I enjoy across-the-table communication latency :-)

[ndmitchell]

It's not going to be on that timetable I'm afraid. I would still be tempted to leave it - once the new stuff is in Shake wiring it in should take seconds. Without the new stuff, you have to implement it all yourself, which will be a pain.

[snowleopard]

Sure, we better wait for you. We have plenty of things to hack on in the meanwhile.

[Nadrieril]

There is now a --skip flag in shake, but it works in a surprising way (for me): I have A that depends on B, and I want to hack on B, so I modify B manually and then run ./shake --skip B. The problem is that shake doesn't do anything, whereas I was expecting it to rebuild A using the modified B... Is there a ways to get this behaviour ?
Moreover, ./shake --skip B seems to ignore an alwaysRerun on A.