Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can one increase number of partitions and hence spark nodes used? #339

Closed
lrhazi opened this issue Dec 11, 2014 · 2 comments
Closed

Can one increase number of partitions and hence spark nodes used? #339

lrhazi opened this issue Dec 11, 2014 · 2 comments
Labels
question :Spark v2.0.3 v2.1.0.Beta4

Comments

@lrhazi
Copy link

lrhazi commented Dec 11, 2014

I am not sure yet how is one supposed to use Spark to process data in ES... but running my first tests, I note that it is using three partitions, while I have more nodes in the spark cluster. seems the number three is from the number of shards in the index, and more generally maybe it is the number of shards used in the query? anyways... is there a way to make it use more nodes? or is that not really useful thing to do?

Thanks a lot,
Mohamed.
@costin
Copy link
Member

costin commented Dec 11, 2014

@lrhazi have you looked into the architecture section in the reference documentation? You're assessment is right that the number of tasks is dictated by the number of shards on the ES front and increasing the tasks beyond this number is not possible since it would simply duplicate the work the other workers are doing.
Each task should work on its own 'slice' of data - allocating multiple workers on the same slice means the slice itself needs to be further divided but that's not really efficient ...

Cheers,

P.S. This aside, please use the mailing list or IRC for discussions/questions instead of the issue tracker.

@lrhazi
Copy link
Author

lrhazi commented Dec 11, 2014

Cool. thanks a lot.

@lrhazi lrhazi closed this as completed Dec 11, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question :Spark v2.0.3 v2.1.0.Beta4
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@costin @lrhazi