[exec,lib] partial revert of using main_session for KlioConfig #147
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DanSimon
force-pushed
the
dsimon/undo-exec-main-session-config
branch
from
410eb84
to
192f5b5
Compare
DanSimon
force-pushed
the
dsimon/undo-exec-main-session-config
branch
from
192f5b5
to
a57879c
Compare
DanSimon
commented
Jan 14, 2021
| # TODO : remove this when the call in spotify-klio is removed | ||
| def _compare_runtime_to_buildtime_config(runtime_config_path): | ||
| return True | ||
| def _compare_runtime_to_buildtime_config(klio_config): |
There was a problem hiding this comment.
2 changes from how this function used to work:
- It used to just return
Trueif it couldn't find the build config, apparently to be compatible with old versions that didn't rely on it. I've removed this. - The check used to just hash files, but since the temp-written config file from
klio-clican be slightly different from what it reads in, it was easier to just create a secondKlioConfigand compare their dicts.
| _thread_local = threading.local() | ||
|
|
||
| @classmethod | ||
| def _load_config_from_file(cls): |
There was a problem hiding this comment.
Instead of doing a straight revert, I opted to keep RunConfig around and move this method into it, which will hopefully better encapsulate subsequent changes we make to this logic.
DanSimon
changed the title
econchick
added a commit
that referenced
this pull request
Jan 25, 2021
This was missed when doing PR #147
3 tasks
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a partial revert for f58caf7.
In some cases for unknown reasons,
RunConfig.get()will begin raising an exception that the config is no longer set. This should not be happening, and in some instances the problem doesn't show up until a streaming job has been running for hours, but there is likely some underlying issue in Beam/Dataflow that is leading the main session variable to be deleted.This PR is not intended to be a complete fix, but rather a mitigation that reverts back to
klioexecgetting its config via theeffective-klio-job.yamlthat should be included with the packaged code.I've left the
klio-clichanges untouched, meaning it will continue to write a temp file and bind it for use byklioexec.klioexecwill also still set theKlioConfigto the main session variable withRunConfig.set(), even though for now it won't be used.Checklist for PR author(s)
[cli] Fixes bugs in 'klio job fake-cmd'.docs/src.