New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError in dask-jobqueue/core.py when starting up workers (v0.4.0) #154

Closed
kaelancotter opened this Issue Sep 18, 2018 · 9 comments

Comments

Projects
None yet
3 participants
@kaelancotter
Copy link
Contributor

kaelancotter commented Sep 18, 2018

Error Seems pretty straightforward. Adding a quick int() fixed things on my end. LMK if you want a PR and or CI coverage... (I didn't see a contributing guide anywhere in the docs) For context grid is an SGECluster object. Thanks for the great tool.

In [6]: grid.scale(2)

DEBUG:Scaling up to 2 workers.
DEBUG:Found 0 active/pending workers.
DEBUG:starting 2 workers
In [7]: tornado.application - ERROR - Exception in callback <functools.partial object at 0x7fba4979c208>
Traceback (most recent call last):
  File ".../miniconda/envs/dask/lib/python2.7/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
  File ".../miniconda/envs/dask/lib/python2.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File ".../miniconda/envs/dask/lib/python2.7/site-packages/dask_jobqueue/core.py", line 375, in scale_up
    self.start_workers(n - active_and_pending)
  File ".../miniconda/envs/dask/lib/python2.7/site-packages/dask_jobqueue/core.py", line 299, in start_workers
    for _ in range(num_jobs):
TypeError: range() integer end argument expected, got float.
@kaelancotter

This comment has been minimized.

Copy link
Contributor Author

kaelancotter commented Sep 18, 2018

Looking a little deeper, its certainly a python 2.7 vs 3.x issue specifically https://www.python.org/dev/peps/pep-3141/

@mrocklin

This comment has been minimized.

Copy link
Member

mrocklin commented Sep 18, 2018

Thanks for the bug report @kaelancotter . A pull request would be welcome. Contributor guidelines are available at documentation for the parent project: http://dask.pydata.org/en/latest/develop.html

@guillaumeeb

This comment has been minimized.

Copy link
Member

guillaumeeb commented Sep 20, 2018

CI complement and fix in a PR would be quite welcome @kaelancotter. Currently we only run "Real" python 2.7 tests with SGE Scheduler, you could just add a test using scale instead of start_workers here.

Should we also add a link to contributing guide somewhere?

@kaelancotter

This comment has been minimized.

Copy link
Contributor Author

kaelancotter commented Sep 20, 2018

I took a look through the travis ci logs for python 2.7 sge. Right now it looks like the sge_master image in the compose stack that executes pytest uses a default 3.x python from miniconda3 instead of the travis_python_version. I think fixing the container to use the travis python version will greatly improve 2.7 test coverage, including coverage of the original type error. I hipe Im not overstepping. :)

@guillaumeeb

This comment has been minimized.

Copy link
Member

guillaumeeb commented Sep 20, 2018

You're probably right, thanks for raising this too! It seems currently python 2.7 sge tests are useless.

It's really nice having other people stepping in this code, so you're very welcome here!

Feel free to submit one or two separate PR, for the initialy raised problem, and for test coverage which seems more complicate to fix.

kaelancotter pushed a commit to kaelancotter/dask-jobqueue that referenced this issue Sep 21, 2018

kaelancotter pushed a commit to kaelancotter/dask-jobqueue that referenced this issue Sep 22, 2018

guillaumeeb added a commit that referenced this issue Sep 22, 2018

Fix issue #154 (#155)
 Fix issue #154, num_job was a float in py2.
@kaelancotter

This comment has been minimized.

Copy link
Contributor Author

kaelancotter commented Sep 22, 2018

Awesome Thanks @guillaumeeb !

@guillaumeeb

This comment has been minimized.

Copy link
Member

guillaumeeb commented Sep 22, 2018

Thanks to you for your help here!

guillaumeeb added a commit that referenced this issue Oct 2, 2018

Pass python version to docker container, see issue #154 (#157)
Pass python version to docker container, see issue #154
@guillaumeeb

This comment has been minimized.

Copy link
Member

guillaumeeb commented Oct 2, 2018

Closing thanks to #155 #158 and #157

@guillaumeeb guillaumeeb closed this Oct 2, 2018

@guillaumeeb

This comment has been minimized.

Copy link
Member

guillaumeeb commented Oct 2, 2018

Thanks again @kaelancotter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment