-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tinybase: New Input/Output, Renamed Executors, Storage/Directory brokering by Project #949
Conversation
Also generalize it so that it can take any object that follows the concurrent.futures interface
This required some rearchitecting of some of the nodes, but in particular made SeriesNodes much nicer
Already done in the interface.
Like parallelization parameters, run time limits and working directory.
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
@liamhuber The details of the bug I mentioned are described here. Since in the snippet we discussed you used getstate/setstate on your own outside of h5io, it might work for you anyway though. |
The error now is related to pip/setuptools, googling around this can be caused when some dependencies are not compatible to the latest version of setuptools. In fact downgrading it seems to resolve the issue locally for me, but no idea what "not compatible" actually means in this context and how to fix it for the CI. @liamhuber I should hope that if you just check this out locally and install the dependencies with conda/mamba it should work. :| |
Question: Why are the windows tests so slow? Windows takes 12 minutes and counting, while other OSs take <5 minutes. I compared the windows log to a couple ubuntu logs, and the actual tests themselves (55 OK + 1 skipped in all cases) do take longer, but we're talking 8s instead of 5s -- insignificant. I didn't see where the actual time is being eaten, but it must be somewhere in the env setup since the tests themselves are OK. Ultimately, I find this acceptable. The windows tests are annoyingly slow, but it's something like 3x not 10x, and the windows operation times are not horrible (at least in what is unit tested). It's a bit puzzling, but I can live with it. |
Omitting the ASE notebook over on #963 did indeed allow the rest of the notebook tests to complete ok, and quickly. The Basic, Lammps, and Shell notebooks all fail exactly as shown above. Most of these look to me like they're just hiccups in either the code or example calls, and that providing missing data, arguments, etc. will clean things up. Interestingly, the TinyBase notebook ran totally fine on the CI. I don't have any knee-jerk response to the error message I got and am not sure why this is. It is also not high-priority for me to figure it out. Regarding the ASE notebook, the pickling error above smells a lot like pickle's difficulties with serializing stuff defined inside a jupyter notebook. @pmrv, I don't plan to patch any of these problems imminently, but if I do get to it then it will be in a stacked PR and I'll open it as a draft early in the process so we don't duplicate any work. In the meantime I don't want to migrate the submodule from here to the workflows while the demos are broken. |
Thanks for the detailed check! Sounds like I was careless and only checked the working directory changes on TinyJob.ipynb. Will update next week when we're back. |
We previously set the AbstractTask.context.working_directory in execute automatically, if not set before. But this can break re-usability of a task, so do not do it for now.
Input Notebook: notebooks/tinybase/Shell.ipynb
|
|
This was because the shell task now requires the |
The `ShellTask` now requires working directory input, and I un-hardcoded the resource path.
…nydata_notebook_patch
Update shell notebook
Supersedes #948 and #936.
SeriesTask
and makes it e.g. possible to run Murnaghan calculations with static, minimizing and molecular dynamics quite generically.Submitter
s now and work with anything that follows theconcurrent.futures
interface. In the future I'll likely completely remove them.