Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: allow indexers to have workspace #1383

Merged
merged 1 commit into from
Dec 2, 2020

Conversation

cristianmtr
Copy link
Contributor

@cristianmtr cristianmtr commented Dec 2, 2020

We want to allow setting of workspace in Indexers without name.

Brought up in #1380 , when tried setting meta={'workspace': tmpdir} but I get an error

like so

with BinaryPbIndexer('pbdix.bin', metas={'workspace': tmpdir.strpath}) as idxer:
(<class 'AttributeError'>, AttributeError("'BinaryPbIndexer' object has no attribute 'warn_unnamed'"), <traceback object at 0x7f243731fec0>)

Copying the approach from here worked https://github.com/jina-ai/jina/blob/008638d3e61c20c585b376ea003be712123a6eb2/tests/unit/executors/indexers/test_numpyindexer.py#L39

metas = {
        'is_trained': False,
        'is_updated': False,
        'batch_size': None,
        'workspace': f'{tmpdir}',
        'name': None,
        'on_gpu': False,
        'warn_unnamed': False,
        'max_snapshot': 0,
        'py_modules': None,
        'pea_id': '{root.metas.pea_id}',
        'separated_workspace': '{root.metas.separated_workspace}',
    }
    # FIXME how to open a specific file in a dir. os.path.join(tmpdir, file) doesn't work
    with BinaryPbIndexer('pbidx', metas=metas) as idxer:

Now the idx files are stored in the tmpdir, instead of the curdir.

Seems like the issue is with 'warn_unnamed': False,

@codecov
Copy link

codecov bot commented Dec 2, 2020

Codecov Report

Merging #1383 (fe599ab) into master (be1d5b7) will increase coverage by 0.04%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1383      +/-   ##
==========================================
+ Coverage   83.45%   83.49%   +0.04%     
==========================================
  Files         103      103              
  Lines        6792     6792              
==========================================
+ Hits         5668     5671       +3     
+ Misses       1124     1121       -3     
Impacted Files Coverage Δ
jina/drivers/querylang/select.py 79.31% <0.00%> (-6.90%) ⬇️
jina/peapods/grpc_asyncio.py 80.61% <0.00%> (+5.10%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b57983...fe599ab. Read the comment docs.

@hanxiao hanxiao merged commit b598437 into master Dec 2, 2020
@hanxiao hanxiao deleted the fix-allow-workspace-in-indexer branch December 2, 2020 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/core This issue/PR affects the core codebase component/executor executor/meta size/XS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants