Added action to ingest file

[johnsca]

No description provided.

[johnsca]

I couldn't get brace expansion to work inside the action without some form of eval, so it just accepts a space-separated list, which lets you do the expansion locally:

juju action do namenode/0 ingest url="$(echo http://example.com/file{1..5}.tar.gz)"

[c0s]

I think a list (or an array) of URLs would suffice. Afterall, the githubarchive API's might be quite unique anyway. I don't know how difficult it might be, but perhaps an alternative form would be to provide a URL to a file, containing the list in it? This way a user can do:

wget --spider http://example.com/file{1..5}.tar.gz > input-list

which might reduce the amount of input significantly

[kwmonroe]

Yeah, i like the idea of taking a list too. Shouldn't be too hard to support in :

dest_dir=$(action-get dest_dir)
url=$(action-get url)

wget_cmd="wget"
test -r $url && wget_cmd="wget -i"

hadoop fs -mkdir -p "$dest_dir"
for url in ${url}; do
    ${wget_cmd} blah foo baz
done

edit: this won't work.. this would look for a file of URLs on the machine where you run "juju action do". That file ain't gonna get over to the unit that actually does the action.

[johnsca]

Per discussion on IRC, I'm now concerned that these downloads can potentially take a fair amount of time, especially if multiple URLs are provided. That would make the action take a long time and block the hook execution queue. We should investigate a way to kick these off in the background and check the status with a separate action.

[c0s]

Running asynchronous actions would solve (or at least partially solve this problem). The ingest can be started right after the storage layer is ready, and continue through the rest of the stack deployment, without interfering with it.

[c0s]

It's all good, I believe. Should be merged, IMHO

[kwmonroe]

LGTM