Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

See if we can speedup postclassical in case of tiling #9493

Closed
micheles opened this issue Feb 28, 2024 · 1 comment · Fixed by #9503
Closed

See if we can speedup postclassical in case of tiling #9493

micheles opened this issue Feb 28, 2024 · 1 comment · Fixed by #9503
Assignees
Milestone

Comments

@micheles
Copy link
Contributor

micheles commented Feb 28, 2024

It is quite slow for the USA model, see calculation #108 on the spot machine:

| calc_108, maxmem=345.0 GB  | time_sec  | memory_mb | counts      |
| total postclassical        | 434_923   | 1_318     | 192         |
| read PoEs                  | 426_713   | 1_318     | 192         |

Received 192 * 1.77 MB {'hcurves-stats': '313.55 MB', 'hmaps-stats': '25.96 MB'} in 4120 seconds from postclassical

[#108 USA INFO] Stored 212.86 GB of rates
[#108 USA INFO] Producing 192 postclassical tasks
[#108 USA INFO] There are 414_311 slices of poes [2157.9 per task]

All process are waiting (in "D" state) while reading the data. A strategy could be to store a tile column and a dataset slice_by_tile rather than slice_by_sid.

@micheles micheles added this to the Engine 3.20.0 milestone Feb 28, 2024
@micheles micheles self-assigned this Feb 28, 2024
@micheles
Copy link
Contributor Author

micheles commented Mar 8, 2024

Solved by gzipping the datasets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant