Hasadna instance max upload size

[akariv]

This task keeps failing, and I suspect that it's related to the max upload size for resources.
The size of the file I'm trying to upload is 1,359,926,398 bytes...
Can you confirm?

https://ckan-dgp.datacity.org.il/en/status/budgetkey-contract-spending-5fa9267a

[OriHoch]

at the moment it seems max upload size is 100mb

I can increase it, but I think having such large datasets is a big strain on the systems, especially the datastore and the automation tasks, so maybe we can allow it but limit the processing of these files

[akariv]

I also saw that 100mb limit but on the other hand, I did upload a 327mb file so not sure if that's the correct number. Some of the hasadna datasets are going to get quite big (especially if we're going to bring data from 250 municipalities), so I think we should find the way to work with these large files which also maintains their usability.

…

On Wed, Nov 11, 2020 at 2:26 PM Ori Hoch ***@***.***> wrote: at the moment it seems max upload size is 100mb I can increase it, but I think having such large datasets is a big strain on the systems, especially the datastore and the automation tasks, so maybe we can allow it but limit the processing of these files — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#127 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACAY5J6Q5SYXVDJ6RT6PXTSPJ7HTANCNFSM4TPTGE2A> .

[niryariv]

@akariv wouldn't the budget files be separate per municipality?

[akariv]

Probably not, as we want to aggregate them to be useful.

[niryariv]

continued in #131