Binary pathing doesn't work for mixed builds

[brookesargent]

From #99:

The home directory varies across CI docker images. In CircleCI-provided convenience images, the user is circleci and therefore the home directory is /home/circleci. However, on many docker images, the user might be root and use /root as a home directory.

The problem is that the orb currently uses ~ in all its paths, even for commands that may run on different images. As a result, download_be_executables and save_be_cache write binaries to and cache them from ~/project/bin. Later, commands that try to use these binaries, like with_job_span, also look in ~/project/bin. But if the home directory differs between the image that cached the binaries and the one that restores the cache and tries to use them, the latter will fail to locate the binaries. For example, if the image that tries to use the binaries uses the root user, I observe the following error:

/bin/bash: line 8: /root/project/bin/be-linux-aarch64/buildevents: No such file or directory

[roblundiehill-plutoflume]

I came here to post exactly the same issue :-)

I got it working locally by switching all occurrences of ~/project/bin with /tmp/honeycomb-buildevents which works fine for mixed executors. I guess there may be a preferable path to use than /tmp

[jharley]

@roblundiehill-plutoflume we'd like to look into the option to specify the path (directory or full path -- not sure) with a good default value. Would that be suitable for your needs?

[ravron]

I authored the original #99. An option to specify the caching path would be useful. I see a few options that are not mutually exclusive:

• Choose a different default absolute path that does not include ~. @roblundiehill-plutoflume's /tmp suggestion is not bad. /usr/bin might be a good option too.
• Provide an option to specify the path. Ideally it would be another environment variable.
• Use a relative path. This has the advantage that CCI ensures the working directory is always valid, so you can download the files to e.g. .honeycomb/bin, caching and restoring that relative path.
• Add a step to the restore_be_cache command that checks for the presence of the binary that was hopefully just restored. If the binary is absent, run download_be_executables and save_be_cache again. This would ensure the job could run, at the cost of slower startup on workflows that mix executor types, the first time a new executor type is used.

Unsurprisingly, I noticed that the CCI docs contain this warning:

Caching files between different executors, for example, between Docker and machine, Linux, Windows or macOS, or CircleCI image and non-CircleCI image, can result in file permissions and path errors. These errors are often caused by missing users, users with different UIDs, and missing paths. Use extra care when caching files in these cases.