allow programmatic overridding of yaml config

[cristianmtr]

Atm we have some tests in core that create a workspace in hardcoded paths (based on resources), like

jina/tests/integration/crud/simple/test_crud_readme.py

Line 8 in fd01fd2

def test_crud_in_readme(mocker):

This leads to tests failing and time wasted trying to debug it until you realize that it was reusing the workspace from an older local run. In CI we don't have this problem. We should not have tests that do this, I propose.

The problem is that we are using this _index.yml from resources which has a hardcoded path. Is there any way to override the metas of the indexer programatically in Python?

[FionnD]

Following the refinement meeting today, there needs to be a design discussion for how we solve this ticket.

Specifically focusing on how we solve the issue _index issue. This needs some creative thinking on how we add the workspaces.

I have set up a 30-minute meeting to discuss solutions. To make the meeting as quick as possible, if you have thoughts beforehand please post them here ⬇️

[cristianmtr]

Findings:

Passing kwargs directly to .add doesn't work:

    with Flow().add(uses='_index',
                    **{'name': 'ws_from_code'}
                    ) as f:
        f.index(docs, on_done=mock)

In .add the name is picked up properly. Below pod.name is "ws_from_code".

        pod = BasePod(args, needs=needs)
        op_flow._pod_nodes[pod_name] = pod

Also, the name is kept also in the BasePea and ZEDRuntime classes.

The problem seems to be that if you provide a uses= it doesn't maintain this name. The name is only for the ZEDRuntime and BasePea, not for the ._executor in the runtime.

See here:

jina/jina/peapods/runtimes/zmq/zed.py

Line 64 in 1f4ded9

self._executor = BaseExecutor.load_config(self.args.uses,

[cristianmtr]

#2162 partially fixes this for one specific test. We can use this approach on a per-test basis.

However, we still want in the future to be able to override / specify context args / env. vars. from Python to override any YAML config. Should keep this issue open

[JoanFM]

Closing this issue, as landscape for executors and configuration will heavily change

[cristianmtr]

Closing this issue, as landscape for executors and configuration will heavily change

Will that address this issue?

[JoanFM]

Closing this issue, as landscape for executors and configuration will heavily change

Will that address this issue?

I would reopen later given the new semantics, since quite likely all these observations will be outdated

[cristianmtr]

Closing this issue, as landscape for executors and configuration will heavily change

Will that address this issue?

I would reopen later given the new semantics, since quite likely all these observations will be outdated

But the underlying point is still valid

[JoanFM]

ok let's reopen then

[cristianmtr]

@JoanFM we can do this now with override_with and override_metas, right?

[JoanFM]

yes @cristianmtr