Custom operators

[maowtm]

I discussed this with @mechtaev in chat before we decided to put everything on GitHub so it's more visible. Here is a recall of the basic idea...

Previously it was decided that operators would be implemented as "start/end" internal predicates, so

foo :- (a, b)::merge

will become

foo :- merge_begin(...), a, b, merge_end(...)

when translating to the IR, where merge_begin and merge_end would actually not be accessible to the user.

However when implementing it I thought that there is really no reason why we have to forbid any manual usage of these "intrinsics", since they do not break any datalog logic. We could have a defined way to name and use them, for example _predicate_merge_begin(...end) which both takes an "ID" argument to indicate paring, and any ::merge would be translated into those. They will be "intrinsics" in the sense that "run" is an intrinsic - the SLD solver just treats them as always true (as long as it is fully grounded), and they are really only meant to be interpreted by any image generation code.

However, by allowing user access to this "special" namespace, we could allow them to define their own operators. Consider the following Modusfile:

build_project :-
  (cc("a.cpp"), cc("b.cpp"))::with_std("c++14"),
  (cc("c.cpp"), cc("d.cpp"))::with_std("c++17").

_operator_with_std_begin(id, std) :- _operator_env_begin(id, "CFLAGS", f"-std=${std}").
_operator_with_std_end(id, std) :- _operator_env_end(id, "CFLAGS", f"-std=${std}").

Note that the with_std operator is "defined" in the last two line.

I do admit that this just feels too much like a hack, so maybe it's better to hide this _predicate namespace after all and, if needed, have some other way to define operators.

[maowtm]

Also not sure if we should repeat the arguments in the end call. (i.e. _operator_env_end(id, "key", "value")) For all internal operators this is not needed as these things can be tracked in rust code, but a user writing their own operators may find those useful...

[mechtaev]

@maowtm, thanks for summarizing the discussion. This feels like a macro

$x::with_std(std) => $x::in_env("CFLAGS", f"-std=${std}")

Do we need overlapping regions?

It will be even more interesting if it was possible to define custom cache invalidation conditions. For example, define cache invalidation conditions for git/hg/svn, etc, given some basic intrinsics.

[maowtm]

@mechtaev

This feels like a macro

Yes, and in fact if we don't consider any internal implementation at all what I had is actually quite a bad syntax. Your macro-style definition is much better IMO. But I just proposed this because it is easy to implement and fits into our internal model. A bit of leaky abstraction I guess.

Do we need overlapping regions?

Probably not.

Custom cache invalidation conditions

Not sure how this would work and what would be the use, as usually people would want to rebuild the thing if source has a new commit...?

[mechtaev]

I had a discussion with @barr, and he suggested to use operators that define conditions of cache invalidation. We were thinking of something like that:

• ::no_cache always invalidate the cache
• ::invalidate_cache_if_before(CMD) invalidate cache if CMD, executed before the layer is applied, returns non-exit code.
• ::invalidate_cache_if_after(CMD) invalidate cache if CMD, executed after the layer is applied, returns non-exit code.
• ::invalidate_cache_if_changed(PATH) invalidate cache if the file/directory changes after the layer is applied.

This is just a rough sketch of the API.

[mechtaev]

Or maybe something like ::invalidate_cache_if_different(CMD_BEFORE, CMD_AFTER)