Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Constantly reload Kubernetes config #2869

Merged
merged 2 commits into from
Dec 23, 2019

Conversation

adamnovak
Copy link
Member

This (written on top of #2868) will fix #2867 by reloading the Kubernetes config and making a new API clients every 5 minutes, as suggested in kubernetes-client/python#741 (comment).

We'll still get in trouble if we have a single watch or loop over pods that runs for more than the credential expiration time, but that seems unlikely to happen and would be its own slowness problem anyway.

@adamnovak
Copy link
Member Author

adamnovak commented Nov 21, 2019

@DailyDreaming I got what I think is an unrelated test failure from the Python 2 integration tests:

_ EncryptedAWSJobStoreTest.test_testImportSharedFile__otherCls_awsjobstoretest _
Traceback (most recent call last):
  File "/builds/databiosphere/toil/src/toil/test/jobStores/jobStoreTest.py", line 1061, in setUp
    super(AbstractEncryptedJobStoreTest.Test, self).setUp()
  File "/builds/databiosphere/toil/src/toil/test/jobStores/jobStoreTest.py", line 124, in setUp
    self.jobstore_initialized.initialize(self.config)
  File "/builds/databiosphere/toil/src/toil/jobStores/aws/jobStore.py", line 148, in initialize
    super(AWSJobStore, self).initialize(config)
  File "/builds/databiosphere/toil/src/toil/jobStores/abstractJobStore.py", line 146, in initialize
    self.writeConfig()
  File "/builds/databiosphere/toil/src/toil/jobStores/abstractJobStore.py", line 154, in writeConfig
    pickle.dump(self.__config, fileHandle, pickle.HIGHEST_PROTOCOL)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/builds/databiosphere/toil/src/toil/jobStores/aws/jobStore.py", line 552, in writeSharedFileStream
    yield writable
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/builds/databiosphere/toil/src/toil/jobStores/aws/jobStore.py", line 1122, in uploadStream
    assert bool(self.version) == (self.content is None)
AssertionError

I'm going to rerun the test, but we also might have a bug.

@adamnovak adamnovak force-pushed the issues/2867-constantly-reload-kubernetes branch from 9508391 to 5ac5af6 Compare December 5, 2019 23:57
Copy link
Member

@DailyDreaming DailyDreaming left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@adamnovak adamnovak merged commit 95e7ea3 into master Dec 23, 2019
DailyDreaming pushed a commit that referenced this pull request Feb 12, 2020
* Refresh Kubernetes credentials if we want to talk to Kubernetes and the credentials are even a little old

* Use a with to clearly close our namespace file
DailyDreaming pushed a commit that referenced this pull request Feb 13, 2020
* Refresh Kubernetes credentials if we want to talk to Kubernetes and the credentials are even a little old

* Use a with to clearly close our namespace file
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants