fix: workspace from dump may not match the one from yaml #1756
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jina-bot
added
size/S
area/core
Codecov Report
@@ Coverage Diff @@
## master #1756 +/- ##
==========================================
- Coverage 80.86% 79.38% -1.49%
==========================================
Files 135 135
Lines 6893 6897 +4
==========================================
- Hits 5574 5475 -99
- Misses 1319 1422 +103
Continue to review full report at Codecov.
|
Latency summaryCurrent PR yields:
Breakdown
Backed by latency-tracking. Further commits will update this comment. |
cristianmtr
reviewed
Jan 22, 2021
| @@ -52,6 +52,12 @@ def parse(self, cls: Type['BaseExecutor'], data: Dict) -> 'BaseExecutor': | |||
| if dump_path: | |||
| obj = cls.load(dump_path) | |||
| obj.logger.success(f'restore {cls.__name__} from {dump_path}') | |||
| # consider the case where `dump_path` is not based on `obj.workspace`. This is needed | |||
cristianmtr
approved these changes
Jan 22, 2021
| # consider the case where `dump_path` is not based on `obj.workspace`. This is needed | ||
| # for | ||
| workspace_loaded_from = data.get('metas', {})['workspace'] | ||
| workspace_in_dump = getattr(obj, 'workspace', None) |
There was a problem hiding this comment.
why do we have the scenario that workspace is not defined?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes introduced
After refactoring in #1722, it seems that there was a bug when indexers are moved to docker.
The problem is that when the executor is loaded from the
mapped volumein docker, the workspace is still in the.binand although is capable of recreating itself, it tries to fetch the data from the old workspace.Solution
I propose to have the workspace switched to the
workspacefolder from where is it loaded as it indicates that it has been "moved" or "mapped into thedockersystem".In the long term, I would suggest to get rid of
.binand have a single source of configuration from YAML and have every executor know what to expect