Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Ifremer config. #294

Merged
merged 4 commits into from Jul 22, 2019

Conversation

@lesteve
Copy link
Member

lesteve commented Jul 22, 2019

Fixes #292.

@lanougue here is my attempt to synthetize the information you gave me in #292. Comments more than welcome!

Note that I added the link to the issue if people wants all the gory details ;-)!

@guillaumeeb

This comment has been minimized.

Copy link
Member

guillaumeeb commented Jul 22, 2019

@lesteve

This comment has been minimized.

Copy link
Member Author

lesteve commented Jul 22, 2019

That may be a better place indeed.

Would you know of a link to the Ifremer cluster. I found https://wwz.ifremer.fr/pcdm/Equipement but it is in French and I am not sure this is the right cluster ...

@lesteve lesteve force-pushed the lesteve:add-ifremer-config branch from a8eee6f to 3e99a01 Jul 22, 2019
@guillaumeeb

This comment has been minimized.

Copy link
Member

guillaumeeb commented Jul 22, 2019

Yep this is the good one: Datarmor. I don't see an english published page.

@lesteve lesteve force-pushed the lesteve:add-ifremer-config branch from 3e99a01 to 59bc625 Jul 22, 2019
@lesteve

This comment has been minimized.

Copy link
Member Author

lesteve commented Jul 22, 2019

This is how it looks on my local build of the documentation:

image

@guillaumeeb

This comment has been minimized.

Copy link
Member

guillaumeeb commented Jul 22, 2019

Hm, I think the death_timeout part should be remove, this is normaly solved by local_directory

@lesteve

This comment has been minimized.

Copy link
Member Author

lesteve commented Jul 22, 2019

OK I think I'll remove it since these slightly gory (but potentially useful) details are in the associated github issue.

I have to say that 15 minutes is quite high. I really like to have a better understanding of what is happening at startup on the local disk for things to be that slow ... this is not related to spilling at all right?

@lesteve lesteve force-pushed the lesteve:add-ifremer-config branch from 130944c to f77d7ce Jul 22, 2019
@guillaumeeb

This comment has been minimized.

Copy link
Member

guillaumeeb commented Jul 22, 2019

No, IIRC this was due to the fact that @lanougue was not using disk local to its compute node, but a network shared space instead, which was causing things to be really slow. But it's just Dask workers that create some folders and put locks in it.

@lesteve

This comment has been minimized.

Copy link
Member Author

lesteve commented Jul 22, 2019

Interesting thanks for the details, 15 minutes to create a few folder and files seems quite high still ...

@lesteve

This comment has been minimized.

Copy link
Member Author

lesteve commented Jul 22, 2019

OK I am going to merge this one, thanks a lot for your inputs @lanougue!

@lesteve lesteve merged commit e26d554 into dask:master Jul 22, 2019
1 check was pending
1 check was pending
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
@lesteve lesteve deleted the lesteve:add-ifremer-config branch Jul 22, 2019
@lanougue

This comment has been minimized.

Copy link

lanougue commented Jul 22, 2019

You are welcome. Thanks for the doc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.