upload reads the entire file with readFile before storing, and multi-upload buffers the whole batch — peak memory scales with upload size, which breaks large uploads and batch runs. Files should stream to the provider, with only stat sizes read up front for prepare().
Separately, dataset upload <path> <providerId> duplicates what upload already does (auto-creates a dataset) through a lower-level path with a worse interface. Remove it (breaking change).
uploadreads the entire file withreadFilebefore storing, andmulti-uploadbuffers the whole batch — peak memory scales with upload size, which breaks large uploads and batch runs. Files should stream to the provider, with onlystatsizes read up front forprepare().Separately,
dataset upload <path> <providerId>duplicates whatuploadalready does (auto-creates a dataset) through a lower-level path with a worse interface. Remove it (breaking change).