Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

import_table min_partitions broken? #5603

Closed
konradjk opened this issue Mar 14, 2019 · 4 comments
Closed

import_table min_partitions broken? #5603

konradjk opened this issue Mar 14, 2019 · 4 comments
Assignees

Comments

@konradjk
Copy link
Collaborator

@konradjk konradjk commented Mar 14, 2019

loading a file that has 12 partitions. when setting hl.init(min_block_size=0) and then hl.import_table(..., min_partitions=100), now getting only 3 partitions.... ☹️

@konradjk
Copy link
Collaborator Author

@konradjk konradjk commented Mar 21, 2019

Ok some weirdness. Running on a cluster of 2 nodes (12 cores), I get 12 partitions. With 12 nodes (92 cores), I get 4 partitions. With 22 nodes (172 cores), I get 172 partitions. (This is without a min_block_size FWIW)

@chrisvittal
Copy link
Collaborator

@chrisvittal chrisvittal commented Mar 21, 2019

Do you have the IR with the TextTableReader json?

@konradjk
Copy link
Collaborator Author

@konradjk konradjk commented Mar 21, 2019

Not handy but can grab it next time I run it.

@tpoterba
Copy link
Collaborator

@tpoterba tpoterba commented Mar 26, 2019

can replicate:

/Users/tpoterba/data/variants_out.txt', min_partitions=1000)
2019-03-26 09:13:47 Hail: INFO: Reading table with no type imputation
  Loading column '19:60864:GCAGCCTCAGCACT:G' as type 'str' (type not specified)


In [3]: ht.count()
[Stage 0:>                                                        (0 + 12) / 12]Out[3]: 848893

In [4]: ht.n_partitions()
Out[4]: 12

@tpoterba tpoterba assigned tpoterba and unassigned chrisvittal Mar 26, 2019
tpoterba added a commit to tpoterba/hail that referenced this issue Mar 26, 2019
danking added a commit that referenced this issue Mar 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants