files as dependency

[alperyilmaz]

Hi,
I'm glad to discover just since make is painful to write. However, I still like some dependencies to be actual files. The recipe should run if file exists or has been updated. "File exists" is easy to find a workaround and there are some workarounds for "if file changed" cases such as watchexec or tricks mentioned in #424. But I was still wondering if it's possible to use a file as dependency so that dependees run if file has changed or file is missing?

Just a hypothetical example:

process-data: input.csv
    #!/usr/bin/env Rscript
    library(tidyverse)
    input <- read_csv("input.csv")
    #some analysis
    write_csv("output.csv")

Is it possible that just process-data does not run if input.csv has not changed?

[casey]

I definitely see why this would be useful, and it's a useful feature of Make.

It's tricky though! Let's use a simplified example. In this justfile, I'm using a string dependency to mean "this dependency is actually a path to an input file". Here's a simple recipe that just copies an input file to an output file:

foo: "input.csv"
  cat input.csv > output.csv

However, this isn't yet enough information. just needs to be told what the output file is, because it should only run this recipe if the output file is older than the input file. Here's the same justfile, but with > indicating an output file:

foo: "input.csv" > "output.csv"
  cat input.csv > output.csv

Now just knows the input file and the output file, and can only run the recipe when input.csv is newer than ouput.csv.

There are some gotchas though. If you were using a compiler, and updated the compiler to a new version, just wouldn't know that since the output file depended both on the input file and the installed compiler, it should run the recipe even though, strictly speaking, the output file was fresher than the input file:

foo: "main.c" > "a.out"
  cc main.c

Similarly, if you were using compiler flags that changed, just consider those flags to be part of the "input". For example, if you changed -O3 to -Os in the following justfile:

foo: "main.c" > "a.out"
  cc -O3 main.c

My hunch is that this is just too hard to get right enough for it to be worth adding, and that there would be a confusing long-tail of issues.

It's not a satisfying answer, but you could always create a Makefile and call it from your justfile. I've also used tup which is like Make but a bit saner, so that could be another option. There's also ninja, which I think is not meant to be used directly, but lots of people do anyways.

In order for me to be convinced that this was a good idea, I think someone would have to present a design which was relatively simple, with straightforward limitations that would make sense to users, and didn't look like it would lead to a long-tail of weird issues.

I'll leave this open in the hope that someone comes up with such a design!

[casey]

You could also embed a ninja file inside a justfile, which is actually kind of nice:

ninja:
  #!/usr/bin/env ninja -f
  cflags = -Wall

  rule cc
    command = gcc $cflags -c $in -o $out

  build main.o: cc main.c

[casey]

(Perhaps not super useful though, since you're using a shebang recipe, and I don't think it's possible to embed non-shell commands in ninja.)

[9SMTM6]

This is the reason I'm not really considering just despite otherwise agreeing with what I've seen from its documentation.

Input>output date based caching isn't perfect, but still it's portable, it's easy to understand, and it doesn't rely on some stateful process that won't survive a reboot etc.

It's not as if just is perfect. You mentioned the possibility of an compiler update. Well, considering you call commands just with a name, a simple alias (perhaps not shell alias but still) with different behavior can also break things perfectly fine.

Provided your commands don't crap out with unexpected input, at worst a input>output caching will fail with unexpected error output and you'll have to go back to the only behavior currently officially supported, just (is that where the name is from?) running everything.

I agree that the default behavior and often the syntax of make often are problematic. But still, that basic behavior is, while not perfect, too valuable for me too give up. Not every command reuses old results like eg cargo, and in that case just with properly declared dependencies becomes unusable for many workflows, and the proposed solutions, just using another build tool, or keeping a watch process running, are not sufficient IMO and have more issues than when input>output would be supported.

So, after that long rant, I'm far from a finished solution, but I believe that idea might be beneficial (though probably would require a lot of work, even if hopefully only additive in behavior).

One could add a way to enable caching given some "was changed" validation returns false. Combined with a helper - provided with appropriate warnings that it's not perfect - that implements the input date>output date validation, and perhaps the easy ability to define other helpers (eg one that checks for changes of used compilers) and combine them, would solve that issue for me and might be acceptable to you.

Where I am seeing most issues is in terms of syntax and in how to handle these validators without introducing a complex system. Perhaps one could define them similar to normal tasks, but still they'd behave differently, as they have to somehow signify true/false.

[casey]

I'm open to this being implemented, but it's a lot of work, and there are a lot of tricky open questions. Make is a kitchen sink of weird features needed to support edge cases, and many aspects of its design are widely regard as very bad.

The best way forward is for someone to figure out how to make just work with redo. Redo is a simple make-like build system with a few implementations, and which has a number of nice properties.

I suspect that just + redo can be made to work relatively easily, and with minimal changes to just.

[runeimp]

I believe Redo, Ninja and others keep a hash for source files and when processing is called a check against the cached hash and the current source files hash is checked to know if it's been modified since the last processing run.

I created a standalone tool for allowing shell scripts and such to be able to have the same functionality with https://github.com/runeimp/filedelta . This sort of functionality obviates the need to check if the source is newer than the target and so only the source needs to be checked, which is handy for dependency checking. And still works if the target is accidentally "touched" after the last modification to the source. The basic process is pretty simple and could be extended to take "context" hash or string as well in cases where the source file may not have changed but needs to be run in a new context.

[naps62]

Anytime I use a Justfile for a project, I end up limited about this issue.
Don't get me wrong, the tool is great, but I think this is really something worth considering more deeply, despite all the caveats already mentioned. So let me try to provide some input

The only concrete syntax proposed here was:

foo: "input.csv" > "output.csv"
  cat input.csv > output.csv

which I'm personally not a fan of. For a few reasons:

• in my opinion, makes the rules much less readable
• it ends up mixing up with the syntax for dependencies on other recipes (test: build). will this lead to the need for PHONY feature as well?
• how would it even play out with the && operator?

Attributes syntax

Looking at the existing syntax already supported by Just, I think a good place to think about this would be Recipe Attributes. A quick (and probably bad) example:

[if(updated(input.csv))]
foo:
  cat input.csv > output.csv

I disagree with @casey that the output file is needed info for this to work. We can choose to compute a hash for the input, store it somewhere, and run only once it changes.
We can also provide something like an --always flag which could be a good quality-of-life feature to skip all these conditions explicity and do a full run after a compiler update.

Custom user commands

As an alternative, or maybe even an addition, we can also opens up the possibility of people specifying their own arbitrary conditions:

needs_build := `find -name input.csv -newer output.csv`

[if(needs_compilation)]
build:

[xonixx]

Just can consider incorporating an approach from makesure.
(full disclosure, I'm the author).

Makesure has @reached_if directive that can be added to any goal. This allows skipping goal execution if it's already satisfied.

Thus you can easily use bash's -nt (newer than) / -ot (older than) test operators like so:

@goal regenerate_file2_based_on_file1
@reached_if [[ file2 -nt file1 ]]
	do_something file1 > file2

Here is an example from real project:

@goal inventory_generated
@depends_on output_ready__instance_public_ip
@reached_if [[ -f "$INVENTORY" ]] && [[ "$INVENTORY" -nt 'inventory.tpl.yml' ]]
@doc 'inventory-<ENV>.yml generated from terraform data'
  instance_public_ip="$(cat "$INSTANCE_PUBLIC_IP")"
  awk -v instance_public_ip="$instance_public_ip" '
{
  gsub(/\$instance_public_ip/, instance_public_ip)
  gsub(/\$ENV/, ENVIRON["ENV"])
  print
}
' inventory.tpl.yml > "$INVENTORY"

Obvious cons of this approach: -nt/-ot are bash-specific, and not present in POSIX shell, and I know nothing of powershell.
Pros: the @reached_id approach is more generic (and so powerful) than just this case since it can evaluate any scriptlet as a condition check.

[arve0]

I disagree with @casey that the output file is needed info for this to work. We can choose to compute a hash for the input, store it somewhere, and run only once it changes.

I like this idea.

What’s the benefit of storing a hash vs the last modified time?

One could also store checksums for the justfile task, to check if you’ve changed it since last run. Storing modified times for binaries used should also be possible.

[runeimp]

The main benefit for hashing is simply that it doesn't require access to the target file to determine if the source changed. This can be important on certain file systems where modification times are beyond your ability to guarantee are stable. This is rarely the case but when it is hashing the source can guarantee a needed build (or whatever) happens.

This can also be important when building the source is incredibly costly in some way and definitely only want to rebuild (or whatever) when the source has definitely changed as apposed to someone opening the source to review, make NO changes, and accidentally saving it out of habit, thus changing it's modification time to newer than the target.

[tgross35]

I think it could be nice to use a functionlike syntax to indicate source files. This way it is unambiguous - you don't have questions about whether your shell variable gets treated as a string, and you can mix and match file targets with other targets:

othertarget:
    echo done

foo: files(a.txt, b.c, d.csv, $foo_file) othertarget > files(x.o)
    cc ...

Or, just create something that is like the opposite of .PHONY in makefiles. Often this kind of usage requires wildcards in the target, which I do not think is currently supported (probably need some form of wildcards for any kind of file-based targets regardless of syntax).

[file]
%.c:

[file]
%.o: %.c
    gcc ...

# Maybe regex could be used? Much more flexible and well known than Make's `%` 
[file]
.*\.o: $1\.c:
    gcc ...

Doing these sort of things with Make is always horrid because of how easy it is to mistype a pattern and get "no rules to make target" somewhere in the graph. Hopefully Just could support something better.

[nmay231]

I think I discovered the simplest solution with the most power:

Add a recipe attribute [cached] that only reruns when the content of the recipe changes, where the content is checked after just-variables are expanded! (or if the last recipe run fails)

It's simple to understand with no extra syntax introduced.

Why is this so powerful? Because it gives access to EVERY function you can use in just

• sha256_file() (and the upcoming blake3_file())
• path_exists()
• we can compare build tool versions with build-tool --version
• we can add a last_modified() and/or is_older_than(a, b) function(s) to support that work flow
• we can also make sha256_file() take a directory or a glob to specify recursive hashing, etc.
• we can use env_var() if is used by the commands in the recipe implicitly.

How do we include these "change detectors" in the recipe content without changing what runs? Put it in a comment.

[cached]
build:
   @# {{sha256_file("input.c")}} Prefix with @ to not print the comment, or don't if, for example, you want to see the build-tool version
   gcc input.c -o output

(We can also add something in the rare case a #! shebang process doesn't have a syntax for comments)

The only weakness of this strategy is when recipe arguments change what is done, not how it is done. For example:

[cached]
build BIN *FLAGS:
   RUSTFLAGS="{{FLAGS}}" cargo build --release --bin {{BIN}}

In the example, building binary1, then binary2 does not mean you need to rebuild binary1 (unless there is some weird dependency between them). However, changing FLAGS certainly means you have to rebuild any previous binary.

The only way to solve it I think is to allow the [cached] attribute to provide ways to "expand" the cache key. For example:

[cached("BIN")]
build BIN *FLAGS:

will evaluate {{BIN}} and include that as part of the cache key, along with the recipe name and project directory absolute path.

I'm gonna start implementing this if there are no major objections and after I finish some other open-source work I've been doing.