feat: pass arbitrary kwargs to docker sdk

[JoanFM]

Changes introduced
As requested in #1671 , adding arbitrary parameters can be useful to get custom behavior or options for the docker images supporting Peas living in Docker images

[tadejsv]

Oh it was this easy 😂

[JoanFM]

Oh it was this easy

Have to test still, so not sure it was this easy, just a draft

[codecov]

Codecov Report

Merging #1690 (0c3e633) into master (122fdf4) will increase coverage by 0.62%.
The diff coverage is 81.06%.

@@            Coverage Diff             @@
##           master    #1690      +/-   ##
==========================================
+ Coverage   84.45%   85.07%   +0.62%     
==========================================
  Files         132      133       +1     
  Lines        6830     6848      +18     
==========================================
+ Hits         5768     5826      +58     
+ Misses       1062     1022      -40     

Impacted Files	Coverage Δ
jina/docker/checker.py	84.90% <ø> (+2.76%)	⬆️
jina/flow/base.py	87.56% <ø> (ø)
jina/executors/indexers/keyvalue.py	98.68% <66.66%> (-1.32%)	⬇️
jina/docker/hubio.py	70.27% <74.35%> (+7.00%)	⬆️
jina/docker/hubapi/remote.py	75.32% <75.32%> (ø)
jina/parsers/helper.py	51.92% <80.00%> (+5.25%)	⬆️
jina/parsers/hub/__init__.py	65.00% <80.00%> (-0.12%)	⬇️
jina/docker/hubapi/local.py	81.70% <81.70%> (ø)
jina/drivers/control.py	95.08% <85.71%> (-1.35%)	⬇️
jina/drivers/cache.py	94.11% <100.00%> (+0.78%)	⬆️
... and 9 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 122fdf4...ba76a55. Read the comment docs.

[github-actions]

Latency summary

Current PR yields:

• 😶 index QPS at 1594, delta to last 3 avg.: +2%
• 😶 query QPS at 31, delta to last 3 avg.: -3%

Breakdown

Version	Index QPS	Query QPS
current	1594	31
0.9.13	1608	32
0.9.12	1503	31

Backed by latency-tracking. Further commits will update this comment.

[tadejsv]

Would you be able to add a test for using this in a yaml config as well?

[JoanFM]

Would you be able to add a test for using this in a yaml config as well?

I am thinking it won't look very nice, I am trying to see how can I pass direct dictionaries to argparser

[tadejsv]

Would you be able to add a test for using this in a yaml config as well?

I am thinking it won't look very nice, I am trying to see how can I pass direct dictionaries to argparser

I also came across this problem when thinking how to implement this yesterday. I think the way you did it is probably the best way currently - argparser is unfortunately very limited - you could use something like eval, but it's use is discouraged.

A more fundamental solution would require moving away from Argparser, but that would take a long time

[JoanFM]

Would you be able to add a test for using this in a yaml config as well?

I am thinking it won't look very nice, I am trying to see how can I pass direct dictionaries to argparser

I also came across this problem when thinking how to implement this yesterday. I think the way you did it is probably the best way currently - argparser is unfortunately very limited - you could use something like eval, but it's use is discouraged.

A more fundamental solution would require moving away from Argparser, but that would take a long time

I found a quite good solution with a helper we are already using.

To use it in the yaml files, the syntax would be similar to the usage of env in Flow.yml as it can be seen in:

tests/integration/optimizers/flow.yml

[tadejsv]

Would you be able to add a test for using this in a yaml config as well?

I am thinking it won't look very nice, I am trying to see how can I pass direct dictionaries to argparser

I also came across this problem when thinking how to implement this yesterday. I think the way you did it is probably the best way currently - argparser is unfortunately very limited - you could use something like eval, but it's use is discouraged.
A more fundamental solution would require moving away from Argparser, but that would take a long time

I found a quite good solution with a helper we are already using.

To use it in the yaml files, the syntax would be similar to the usage of env in Flow.yml as it can be seen in:

tests/integration/optimizers/flow.yml

Pretty cool :) I guess for now it is not possible to pass list/dict arguments, right? I mean the following [but through command line/yaml]

docker_kwargs = {'environment': ['VAR1=BAR', 'VAR2=FOO']}

[JoanFM]

Would you be able to add a test for using this in a yaml config as well?

I am thinking it won't look very nice, I am trying to see how can I pass direct dictionaries to argparser

I also came across this problem when thinking how to implement this yesterday. I think the way you did it is probably the best way currently - argparser is unfortunately very limited - you could use something like eval, but it's use is discouraged.
A more fundamental solution would require moving away from Argparser, but that would take a long time

I found a quite good solution with a helper we are already using.
To use it in the yaml files, the syntax would be similar to the usage of env in Flow.yml as it can be seen in:
tests/integration/optimizers/flow.yml

Pretty cool :) I guess for now it is not possible to pass list/dict arguments, right? I mean the following [but through command line/yaml]

docker_kwargs = {'environment': ['VAR1=BAR', 'VAR2=FOO']}

Not sure, should be tested If u want u can add a tesr and do a PR to this branch or I will try to test tmr

[tadejsv]

I can try testing it a bit later yes

[JoanFM]

Hey @tadejsv , I did some changes to allow passing strings and arbitrary values, plus I added a test to showcase and document how it can be passed into a yaml

[tadejsv]

Great work @JoanFM !!