feat: optimizer yaml interface

[maximilianwerk]

This will enable using a .yml file for running the optimizer.

• Added jsonlines parsing to the index_lines and search_lines
  • Please have a look if this is fine. Especially, the change to data_type=DataInputType.AUTO
• Added simple JAML parsing for any objects which don't need special handling.

Regarding the JAML parsing: I am well aware that the current solution is not in line with how Flows, Executors and Drivers are handled. In case the current solution is not wanted due to broken consistency (or consequences which I have not understood yet), we could also write a simple parser, which is used, whenever no other "special" parser is needed. I would be super happy about better informed input here @hanxiao @JoanFM

[codecov]

Codecov Report

Merging #1726 (1e99cd6) into master (e5e3db8) will decrease coverage by 0.22%.
The diff coverage is 90.62%.

@@            Coverage Diff             @@
##           master    #1726      +/-   ##
==========================================
- Coverage   85.55%   85.33%   -0.23%     
==========================================
  Files         134      135       +1     
  Lines        6834     6893      +59     
==========================================
+ Hits         5847     5882      +35     
- Misses        987     1011      +24     

Impacted Files	Coverage Δ
jina/enums.py	95.91% <ø> (ø)
jina/optimizers/discovery.py	80.26% <ø> (ø)
jina/peapods/runtimes/jinad/client.py	72.38% <50.00%> (+0.20%)	⬆️
jina/optimizers/flow_runner.py	84.44% <75.00%> (-15.56%)	⬇️
jina/clients/sugary_io.py	96.36% <100.00%> (+0.53%)	⬆️
jina/executors/decorators.py	91.27% <100.00%> (ø)
jina/jaml/parsers/__init__.py	100.00% <100.00%> (+2.27%)	⬆️
jina/jaml/parsers/default/v1.py	100.00% <100.00%> (ø)
jina/logging/logger.py	82.40% <100.00%> (+0.28%)	⬆️
jina/optimizers/parameters.py	100.00% <100.00%> (+10.22%)	⬆️
... and 21 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad6306a...1e99cd6. Read the comment docs.

[github-actions]

Latency summary

Current PR yields:

• 🐢🐢 index QPS at 1270, delta to last 3 avg.: -27%
• 😶 query QPS at 33, delta to last 3 avg.: +0%

Breakdown

Version	Index QPS	Query QPS
current	1270	33
0.9.18	1749	32

Backed by latency-tracking. Further commits will update this comment.

[JoanFM]

I do not understand why current JAML parser does not work?

[JoanFM]

Like a lot this pattern

[maximilianwerk]

I do not understand why current JAML parser does not work?

Because the current JAML parser works ONLY for classes, which inherit from BaseDriver, BaseFlow or BaseExecutor. See this function. We could have another SimpleParseableBaseClass which has a simple parser and works if inherited from.

But on the other hand, that is exactly, what JAMLCompatible is for and thus instead of having a raise NotImplementedError(f'{cls!r} does not implement YAML parser') in the above mentioned function, we could implement a default simple parser. That would be my preferred solution, but before implementing this, I wanted to double check if there are some obvious no-gos.

[JoanFM]

I do not understand why current JAML parser does not work?

Because the current JAML parser works ONLY for classes, which inherit from BaseDriver, BaseFlow or BaseExecutor. See this function. We could have another SimpleParseableBaseClass which has a simple parser and works if inherited from.

But on the other hand, that is exactly, what JAMLCompatible is for and thus instead of having a raise NotImplementedError(f'{cls!r} does not implement YAML parser') in the above mentioned function, we could implement a default simple parser. That would be my preferred solution, but before implementing this, I wanted to double check if there are some obvious no-gos.

I do not understand why current JAML parser does not work?

Because the current JAML parser works ONLY for classes, which inherit from BaseDriver, BaseFlow or BaseExecutor. See this function. We could have another SimpleParseableBaseClass which has a simple parser and works if inherited from.

But on the other hand, that is exactly, what JAMLCompatible is for and thus instead of having a raise NotImplementedError(f'{cls!r} does not implement YAML parser') in the above mentioned function, we could implement a default simple parser. That would be my preferred solution, but before implementing this, I wanted to double check if there are some obvious no-gos.

What I understand now, is that you do not want a Parser?

It seems that this may be a thoughtful design to guarantee that every JAML interface is backed by a CLI one, so maybe we should go for implementing our parser?

[maximilianwerk]

Since we just need a trivial parser for OptunaOptimizer, FlowRunner and OptimizationParsers and the parser might look the same for all three (for now) I wondered, whether I should just implement a simple DefaultParser. And I think, that is what I will do straight away and then we see if it is a good design or not.

[JoanFM]

LGTM!