Skip to content
This repository has been archived by the owner on Mar 12, 2024. It is now read-only.

Experiment with different file formats on AWS #1

Open
JackKelly opened this issue Apr 29, 2020 · 1 comment
Open

Experiment with different file formats on AWS #1

JackKelly opened this issue Apr 29, 2020 · 1 comment
Assignees
@JackKelly

Comments

@JackKelly
Copy link
Member

JackKelly commented Apr 29, 2020 •
edited

Compare Zarr vs TileDB vs NetCDF. Also try compressing with pbzip2 -5 (which reduces .nat files to 0.2x their original size). Consider:

  • size on disk
  • write speed
  • read speed for queries similar to ones we'd use for the front-end and for ML training.

https://medium.com/informatics-lab/storing-cloud-ready-geoscience-data-with-tiledb-34d454c33055
@JackKelly JackKelly changed the title Experiment with TileDB on AWS Experiment with different file formats on AWS Apr 29, 2020
@JackKelly JackKelly self-assigned this Apr 29, 2020
@JackKelly
Copy link
Member Author

Note to self: @DPeterK at the Informatics Lab is also looking at benchmarking various file formats.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@JackKelly JackKelly

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JackKelly