Prepare for Dask 2.2 release - #303
Conversation
This is just to test against new versions prior to a release upstream
This avoids serialization issues
mrocklin
changed the title
|
OK, this passes with both master and latest release. I've changed the title. I think that this is ready for review. |
|
Hm, this is difficult to review without context... There is little changes here, I suspect this is the minimum to make it work, but that we should rewrite a lot of the code base to comply with SpecCluster. |
Right, the thing to notice here is the two green check CI marks at the end of the list of commits. One is for using git master, and one is for using the current release (See the last commit, which transitions from git master back to current release). If we don't merge this then dask-jobqueue will not be compatible with the next release.
I agree. I'm changing SpecCluster now to include most of the functionality that dask-jobqueue has had to build on its own. Hopefully the rewrite will be very simple once I'm done. I think that that will happen after the Dask 2.2 release though, so we should merge and release this before the rewrite happens. |
|
|
||
| def worker_key(self, worker_state): | ||
| @staticmethod | ||
| def worker_key(worker_state): |
There was a problem hiding this comment.
The method wasn't easy to serialize because the class has some unserializable elements. Fortunately we don't need self here, so I changed this to a staticmethod in order to remove need to serialize the class.
|
|
||
| @property | ||
| def scheduler_comm(self): | ||
| return self.local_cluster.scheduler_comm |
There was a problem hiding this comment.
This is for compatibility with SpecCluster, which some elements, like adaptive, are starting to expect
| try: | ||
| self._adaptive = Adaptive(self.scheduler, self, **self._adaptive_options) | ||
| except Exception: | ||
| self._adaptive = Adaptive(self, **self._adaptive_options) |
There was a problem hiding this comment.
We dropped the need to explicitly pass in the scheduler in recent versions. We will query the scheduler over a network connection instead so that it can be remote in the future.
|
Thanks for doing this! |
2 participants
This is just to test against new versions prior to a release upstream