using a build cache does not prevent always compiling a subset of the files

[dvic]

Elixir and Erlang/OTP versions

1.13.4 + OTP 25.0.3

Operating system

linux

Current behavior

When I start a fresh phoenix project, and set up a Github action with a cache for the _build folder that simply compiles the app, there are three files that get recompiled each time (with no source code changes):

• gettext.ex
• page_view.ex
• layout_view.ex

I investigated a bit and the only thing I could find is the following. Locally, when I run twice the mix compile command, the build folder remains exactly the same (I compute the SHA digest using find ./_build -type f -print0 | sort -z | xargs -0 shasum | shasum). However, on Github, the digest of the build folder changes each time and it's in particular because of one file (everything else remains the same):

./_build/dev/lib/gh_debug/.mix/compile.app_tracer

Does this ring any bells? Is this expected behaviour or potentially a bug in the compiler?

Expected behavior

If no source code changed, and I'm using a cache for the _build folder, I expect that no files need to be recompiled (just like it's locally the case).

[josevalim]

Both Phoenix Views (via __mix_recompile__?) and Gettext uses a custom compilers (except on latest Gettext which also uses __mix_recompile__?). So I would check if the value of those callbacks are changing between builds and, if so, why.

Otherwise, please provide a minimal project that reproduces the error (ideally with no dependencies), so we can dig deeper. Thank you.

[dvic]

Both Phoenix Views (via __mix_recompile__?) and Gettext uses a custom compilers (except on latest Gettext which also uses __mix_recompile__?). So I would check if the value of those callbacks are changing between builds and, if so, why.

Otherwise, please provide a minimal project that reproduces the error (ideally with no dependencies), so we can dig deeper. Thank you.

Here is a reproduction repo of a freshly generated phoenix project: https://github.com/qdentity/gh_debug_repo (see this build for example)

I already suspected gettext, so I removed that completely from the project, then the phoenix view files still got recompiled, haven't dug into its __mix_recompile__? yet. Will try that now.

[dvic]

Just checked (also gettext) and these functions are returning false:

https://github.com/qdentity/gh_debug_repo/runs/7707636800?check_suite_focus=true

[josevalim]

So Elixir is setting those files as stale. Either because their mtime changed or because it thinks a dependency changed. Any way to reproduce this locally?

[josevalim]

Never mind, I know what it is. Both of those projects also use @external_resource and external resource checks are all mtime based.

[dvic]

Never mind, I know what it is. Both of those projects also use @external_resource and external resource checks are all mtime based.

ahhh that explains a lot :D! I assume it's not trivial to make these external resource checks based on a digest?

[josevalim]

It is not hard but the question is if it is a generally safe assumption /cc @marcandre

You could also set the mtimes to the last commit, but then you must ensure the cache is per branch: https://stackoverflow.com/questions/2458042/restore-a-files-modification-time-in-git

[dvic]

I can confirm that indeed restoring mtimes fixes the issue. Thanks 👍

[dvic]

It is not hard but the question is if it is a generally safe assumption

You mean because of the probability of collisions or in general?

[josevalim]

For example, if you are attaching the mtime of a file, or other information such as user/group, you may want to recompile and recompute it always. I also wonder we could run into cases where computing the md5 would be too expensive. For example, I could use an executable as external_resource, to know when it changes because I use it during compilation, but the executable itself is not inlined nor anything.

Although those use cases are unlikely I think.

[marcandre]

I'd lean towards using digest for external_resources too 👍,
Although I don't have the experience to know about potential dependencies on huge files, let's remember that we also compare sizes before computing the digest, so performance issues seem very remote.
Let me know if I should add this to my todo list (after @behaviour). I'll should have more time after my travel finishes in ~two weeks.

[dvic]

For example, if you are attaching the mtime of a file, or other information such as user/group, you may want to recompile and recompute it always. I also wonder we could run into cases where computing the md5 would be too expensive. For example, I could use an executable as external_resource, to know when it changes because I use it during compilation, but the executable itself is not inlined nor anything.

Although those use cases are unlikely I think.

For the cases where user/group matters, for example, we could maybe add support for @external_resource {path, opts}? Then library authors have full control over when the recompilation happens. For other cases I'd also opt for using digest by default.

[josevalim]

Excellent, I have created #12057.