Get the local and remote repo contents cache to work together

[fmeum]

• Also upload to the remote cache when the local cache is in use. The fix is simple but subtle: the logic for the two caches in RepositoryFetchFunction has to be flipped since the Skyframe restart after adding an entry to the local cache meant that the same code path would not be taken again.
• Fix a crash when using both by ensuring that the local repo contents cache uses the file system backing the output base, not the workspace directory:

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@@rules_python+' (requested by nodes 'REPO_FILE:@@rules_python+')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:552)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException: Files are on different filesystems: C:/users/runneradmin/_bazel_runneradmin/ebfu7cpi/external/@rules_python+.marker (on com.google.devtools.build.lib.remote.RemoteExternalOverlayFileSystem@79583b9), C:/Users/runneradmin/.cache/bazel-repo/contents/_trash/26a5feef-bf8c-4326-bf3d-500997c7362e (on com.google.devtools.build.lib.windows.WindowsFileSystem@24180f0f)
	at com.google.devtools.build.lib.vfs.Path.checkSameFileSystem(Path.java:964)
	at com.google.devtools.build.lib.vfs.Path.renameTo(Path.java:630)
	at com.google.devtools.build.lib.vfs.FileSystemUtils.moveFile(FileSystemUtils.java:456)
	at com.google.devtools.build.lib.bazel.repository.cache.LocalRepoContentsCache.moveToCache(LocalRepoContentsCache.java:172)
	at com.google.devtools.build.lib.bazel.repository.RepositoryFetchFunction.compute(RepositoryFetchFunction.java:297)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:471)

[fmeum]

@bazel-io fork 9.0.0

[gemini-code-assist]

Code Review

This pull request effectively addresses a critical issue where Bazel would crash when both local and remote repository content caches were in use. The changes correctly reorder the cache upload logic in RepositoryFetchFunction.java to ensure that the remote cache is updated even when the local cache is enabled. The addition of testLocalRepoContentsCacheInteraction in remote_repo_contents_cache_test.py provides valuable test coverage for this crucial interaction, verifying the fix and improving the robustness of the caching mechanism. The minor refactoring in BlazeDirectories.java and BazelRepositoryModule.java also contributes to overall code consistency.

[Wyverald]

maybe my brain is just fried right now, but I don't understand this PR at all 😅 Could you please add more to the PR description? Why do the code changes you made fix the bug?

[Wyverald]

I don't understand what this comment means. What doesn't actually write source files? Why does that mean we have to use getWorkspace vs getWorkingDirectory? And why are we mentioning getWorkingDirectory at all?

[fmeum]

I added more explanations to the comment. This is indeed very confusing since getWorkspace() returns a Path that shouldn't be used to write to the workspace :-D In the end no one will write to the workspace through this path (e.g. since we forbid the repo contents cache from living under the workspace), but the FileSystems have to match.

[Wyverald]

could you explain a bit why moving this to the top fixes the bug?

[fmeum]

You are right, this is actually pretty subtle. I added a throw as well as a more detailed PR description to clarify the non-obvious control flow.