New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Q] How can I define max-task-threads in a task with "_parallel: true" parameter? #710

Open
jo8937 opened this Issue Jan 5, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@jo8937
Contributor

jo8937 commented Jan 5, 2018

I'm running bigquery task by "for_each" and "_parallel"
but there is so many task, and I want adjust _parallel task concurrent thread number limit to 8
( bigquery's max concurrent query is 50. and I have to use under 10 concurrent bigquery in our company )

@jo8937 jo8937 changed the title from Can I define max-task-threads in a task with "_parallel: true" parameter? to [Q] How can I define max-task-threads in a task with "_parallel: true" parameter? Jan 5, 2018

@hiroyuki-sato

This comment has been minimized.

Contributor

hiroyuki-sato commented Jan 5, 2018

Hello @jo8937

Does this mean you want to limit the number of thread in _parallel like the below?
As far as I know, It's not implemented yet.

_parallel: true
concurrenly: 8

You can limit it using --max-task-threads N parameter.
ex) digdag server --max-task-threads N digdag run --max-task-threads N.

@hiroyuki-sato

This comment has been minimized.

Contributor

hiroyuki-sato commented Jan 5, 2018

if (count > Limits.maxWorkflowTasks()) {
throw new ConfigException("Too many loop subtasks. Limit: " + Limits.maxWorkflowTasks());
}

https://github.com/treasure-data/digdag/blob/master/digdag-core/src/main/java/io/digdag/core/Limits.java#L5-L6

Have you ever tried io.digdag.limits.maxWorkflowTasks system property?
ex) digdag -X io.digdag.limits.maxWorkflowTasks=2 run -a fuga?

@jo8937

This comment has been minimized.

Contributor

jo8937 commented Jan 7, 2018

Oh, yes! I want exactly like this.

_parallel: true
concurrenly: 8

I didn't try to use io.digdag.limits.maxWorkflowTasks parameter.
but I think this parameter applies to whole workflow. not only apply to subtask .
(I've using digdag in server mode with 6 servers)

I hope 'concurrently' parameter implemented soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment