Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support synchronizing with a custom script / executable #146

Closed
dflems opened this issue Aug 16, 2018 · 9 comments · Fixed by #185

Comments

@dflems
Copy link

@dflems dflems commented Aug 16, 2018

Until support exists for more storage systems, the only options for using Rome are S3 and the local cache. For security and other reasons, we can't easily use S3 with our current setup. I propose that Rome adds support for synchronizing using a custom script or executable with a well-defined interface. A status code of 0 would indicate success and standardized error codes could indicate other statuses (file not found, etc). This way, people can plug in the script to add support for whatever storage system they want no matter how bespoke or esoteric it may be:

Supply custom-exec option in Romefile

[Cache]
local = ~/Library/Caches/Rome
custom-exec = ./my-custom-sync-script.sh

rome executes the script to handle actions

# upload to remote
./my-custom-sync-script.sh upload <local-path> <remote-path>

# download from remote
./my-custom-sync-script.sh download <remote-path> <local-path>

# probe remote for existence of file
./my-custom-sync-script.sh probe <remote-path>

Trivial example of a custom HTTP sync script

#!/bin/bash
set -e

ACTION="$1"
BASE_URL="http://my.server.internal/rome_cache"

if [ "$ACTION" == "upload" ]; then
  # make PUT request to save file on remote
  LOCAL_PATH="$2"
  REMOTE_PATH="$3"
  curl -sf -X PUT "$BASE_URL$REMOTE_PATH" -d "@$LOCAL_PATH"

elif [ "$ACTION" == "download" ]; then
  # make GET request for file, save to specified output path
  REMOTE_PATH="$2"
  OUTPUT_PATH="$3"
  curl -sf "$BASE_URL$REMOTE_PATH" -o "$OUTPUT_PATH"

elif [ "$ACTION" == "probe" ]; then
  # make HEAD request to check for existence of file
  REMOTE_PATH="$2"
  curl -sfI "$BASE_URL$REMOTE_PATH" > /dev/null
else
  # unsupported command
  exit 1
fi
@tmspzz

This comment has been minimized.

Copy link
Owner

@tmspzz tmspzz commented Aug 16, 2018

Hello,

thanks for opening an issue.

Until support exists for more storage systems, the only options for using Rome are S3 and the local cache.

You can use Minio or Ceph, or any other S3 compatible store. You don't have to forcibly use S3.

Nevertheless this is a good idea.

Rome supports a variety of switches, including:

  • --platform (restrict operations to the specified platforms)
  • --cache-prefix (a prefix to append to the remote paths)
  • -v (verbose)

I have the feeling these should be passed on, just in case.

I would be happier if the communication between rome and the "engines"
was based on JSON for I/O rather than relying on exit codes.

@tmspzz

This comment has been minimized.

Copy link
Owner

@tmspzz tmspzz commented Aug 16, 2018

For example the scripts can be called with:

  • the current executing command: one of upload , download or list
  • all the parameters rome was called with

as parameters to the script:

eg. ./my-custom-sync-script.sh download --cache-prefix somePrefix -v --platforms ios,mac

the script would then:

  • read from stdin the JSON representing the data
  • write to stdout write output data

This would mean that you might not be able to write random stuff to stdout, or the random lines should be prefixed with some escape sequence like # line of text

@dflems

This comment has been minimized.

Copy link
Author

@dflems dflems commented Aug 16, 2018

Forwarding the original args makes sense, as does stdout/stderr.

You can use Minio or Ceph, or any other S3 compatible store. You don't have to forcibly use S3.

We have a filestore service (sadly not S3 compatible) that is by default read-only on our VPN or office network, and CI machines are credentialed to write to it. That seemed like the ideal producer/consumer flow, but we need something like this script-runner support to handle it.

Another solution could be a transparent pass-through proxy to our internal storage service that emulates the S3 API. That was my original idea but this one seemed a bit cleaner and extensible for other storage solutions, not just my own :)

@dflems

This comment has been minimized.

Copy link
Author

@dflems dflems commented Aug 16, 2018

Another option that could suit our needs (and maybe others) is simple HTTP GET/PUT/HEAD with optional credentials.

In this case, you would provide a base URI, for example https://my.server/base. If you wanted to upload the file foo/bar/baz.zip, the URI you would interact with would then be https://my.server/base/foo/bar/baz.zip.

  • list command would make HEAD request
  • download command would make a GET request
  • upload command would make a PUT request

You can assume normal HTTP status codes indicate the success or failure of the request.

You can optionally supply a credentials file (json or yaml) via configuration or environment variable that indicates that additional headers that the request should be decorated with:

Authorization: my-special-token

In our case, we could use that for uploading content on the CI side.

@tmspzz

This comment has been minimized.

Copy link
Owner

@tmspzz tmspzz commented Aug 16, 2018

Great. Now two questions:

  • Who is going to develop the feature? Can you help? I am busy with #135 atm
  • What company is this for? Is this for your own project?

I'm asking this in case, you know, it's down to me to work for free (yes, of course I care for the adoption of the project but you know, I have a day job too and just so many hours at night)

See my roadmap: https://github.com/blender/Rome/projects/1

Another option that could suit our needs (and maybe others) is simple HTTP GET/PUT/HEAD with optional credentials.

How about .netrc ?

@tmspzz

This comment has been minimized.

Copy link
Owner

@tmspzz tmspzz commented Aug 16, 2018

Actually if all you want is to add an Authorization header then rome should not even care about your credentials. In shorts you could just specify in some file:

headers:
 - Authorization Basic: SGVsbG86dGhlcmUK

and rome will just add this header

@dflems

This comment has been minimized.

Copy link
Author

@dflems dflems commented Aug 23, 2018

@blender: I don't have a lot of time at the moment, but if I can find some free time I can look into doing it. I'd need to learn haskell first though :)

@tmspzz

This comment has been minimized.

Copy link
Owner

@tmspzz tmspzz commented Aug 23, 2018

Leaning Haskell is rewarding in itself ;)

@tmspzz

This comment has been minimized.

Copy link
Owner

@tmspzz tmspzz commented May 18, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Rome-General
  
Done
2 participants
You can’t perform that action at this time.