[Bug]: bk artifacts download is unusably slow

[zenspider]

Contact Details

um. here?

What happened?

(from an email I sent to support@ who sent me here)

I found this to be tedious at best:

bk artifacts ls -p platform | jq -r '.[] | select(.path | test("rspec.*json")) | .id' | xargs -n1 bk artifacts download -p platform

given:

3183 $ bk artifacts ls -p platform | jq -r '.[] | select(.path | test("rspec.*json")) | .id' | wc -l
     70

this is SUPER slow with xargs -n1. It's equally slow(er) not trying to grab specific files. bk build download is incredibly fast by comparison.

PLEASE, either let me specify multiple IDs per cmdline OR allow me to select on the path(s) I want, OR let me get the whole shebang in one fast zip. I would much prefer:

bk artifacts download -p platform 'log/rspec*.json'

or even

bk artifacts download -p platform $(bk artifacts ls -p platform | jq -r '.[] | select(.path | test("rspec.*json")) | .id')

and better/relatedly... I'd rather all the artifacts be included with bk build download since that's actually fast but seemingly unhelpful since it contains TWO of the rspec files in question (I really don't understand that at all and am guessing that I'm not getting some fundamental concept).

I can share more details if it helps.

Version

3.49.2

What environment are you seeing the problem on?

Local Development

Relevant log output

[zenspider]

bk artifacts ls -p platform 305276 | jq -r '.[] | select(.path | test("rspec.*json")) | .id' | xargs -P 5 -n 1 bk artifacts download -p platform -b 305276

note the redundant -b 305276 in the xargs. that was fun to figure out.

even with xargs -P 5, this command took several minutes to run.

And why is it bk artifacts ls <build-number> but bk artifacts download --build <build-number>?

Can you PLEASE acknowledge this issue?

[lizrabuya]

Hi @zenspider, sorry for not responding back to the issue you raised. I had added the support for filtering artifacts by --path on this PR - #934. This feature should come out on the next release.

We will work next on supporting the filter options in bk build download .

For your question on the difference in the usage for bk artifacts list and bk artifacts download, the bk artifacts download has ` as its argument, thus the build number is passed as an optional flag.

[lizrabuya]

Hi @zenspider , we have now extended bk build download to support artifacts filtering with --artifacts-paths in this PR #938. This should complete all the issues that you've raised here.