Storing relative paths in the .cache archive

[wagdav]

First, thanks for making cache-s3. It's a really useful tool!

I'm using cache-s3 on a compilation cache database (created with clcache on Windows). My builds are running on a node in a temporary directory assigned randomly by the CI system. My compilation cache is a subfolder of the working directory.

If I do two consecutive builds, I'm assigned two different working directories, so that the build artifacts won't interfere. But since cache-s3 stores absolute paths I cannot restore the cache from the previous build.

In short, the first build

# working directory is $BUILD_DIR_1, distributed randomly by the CI system

[perform the build]

cache-s3 save -p $BUILD_DIR_1/cache  # will save absolute paths

Now a consecutive build

# working directory is $BUILD_DIR_2, distributed randomly by the CI system

cache-s3 restore # will write into $BUILD_DIR_1/cache instead of $BUILD_DIR_2

[Something is written outside my working directory, but I don't know where?]

What would you suggest in this case? Would it be possible to save the cache content using relative paths?

[wagdav]

I was looking at the code a bit and I'd have two proposal for this.

1. Remove the path normalization and just use the paths as they are given on the command line

cache-s3 save --path my_directory           # would save relative paths
cache-s3 save --path $HOME/my_directory     # would save absolute paths
cache-s3 save --path $HOME/dir1 --path dir2 # dir1 would be absolute, dir2 relative

This would break the current behavior, which always saves absolute paths.

2. Add a new command line option to save

The current behavior could be extended so that

cache-s3 save --path <dir1> --path <dir2> --relative-to <base_dir>

would save the provided path relative to a base directory. The base_dir would apply to all --path arguments. If any of the path is not a subdirectory of the base directory, the absolute path would be used (the actual behavior).

As this is blocking me, I'd be happy to contribute a PR on this. I believe the Option 1 would be easier to implement, but it would be a breaking change. I think it's also straightforward to understand.

Option 2 would be an extension, but I think it makes the program's behavior more complex, for a little gain.

What do you think? Maybe there's a better option?

[lehins]

@wagdav I do like the idea of relative paths, moreover I was sure someone would eventually ask for it, but I didn't implement it initially since there was need for it, but now that there interest in that feature, I'll be happy to accept a PR and provide help with it if necessary.

As far as your suggested options, I would say that I definitely don't like the first one, since it is NOT backwards compatible and could break current deployments, which is no bueno.
Option 2 is indeed a bit too complex and also non-intuitive. Here is what I propose, adding another flag --rel-path, in which case, for simplicity it will save paths that are relative to current directory. Example:

/tmp$ # dir1 absolute, dir2 absolute, dir3 would be relative to /tmp:
/tmp$ cache-s3 save --path $HOME/dir1 --path dir2 --rel-path dir3

Whenever restoring files, all paths that had absolute paths are naturally restored where suppose to and ones with relative paths, would be restored into current directory.

What do you think, does that solution fit your use case?

Just a side note to this solution. It would be impossible to save paths that are relative to different base paths, but nothing prevents a user to invoke cache-s3 multiple times and save them as different .cache files from different directories.

[wagdav]

@lehins I like your proposition, I'll give it a try to code it up.

[lehins]

@wagdav Thanks for the PR, you wrote exactly in a way I envisioned it ;)
I did give it a try and it worked like a charm. New release v0.1.6 is available.

[wagdav]

Thanks @lehins for the fast release! Tomorrow we're deploying the new version 😃