Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel ingestions on the same machine #16

Open
stuarteberg opened this issue Oct 8, 2023 · 0 comments
Open

Parallel ingestions on the same machine #16

stuarteberg opened this issue Oct 8, 2023 · 0 comments

Comments

@stuarteberg
Copy link
Member

stuarteberg commented Oct 8, 2023

We don't customize the port that neo4j uses, so presumably we'll run into conflicts of we attempt to export two different neuprint datasets on the same machine at the same time.

It can't really happen on the Janelia cluster if we require more than half of the slots on any given node. But if we ever reduce our slot/RAM requirements, then this may become an issue.


Update: The Janelia cluster recently added 64-core nodes, so if we only request 32 slots per job, we are at risk of having multiple ingestions run in parallel on the same machine!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant