feat(dataset): external storage backend

[m-alisafaee]

Description

Removes external files. Replaces is_external with linked for linked files. Adds an external storage backend for external files. Store file sizes when adding files to datasets.

TODO

• Set minimum version to 2.4.0 since we add new fields to the metadata.

Fixes #3284
Fixes #3279

[coveralls]

Pull Request Test Coverage Report for Build 4449216907

• 376 of 507 (74.16%) changed or added relevant lines in 34 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+86.0%) to 85.991%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
renku/core/dataset/providers/factory.py	9	10	90.0%
renku/core/dataset/providers/s3.py	14	15	93.33%
renku/core/workflow/activity.py	0	1	0.0%
renku/infrastructure/database.py	5	6	83.33%
renku/infrastructure/storage/factory.py	1	2	50.0%
renku/command/dataset.py	4	6	66.67%
renku/command/format/dataset_files.py	34	36	94.44%
renku/core/dataset/providers/azure.py	17	19	89.47%
renku/core/interface/storage.py	8	10	80.0%
renku/domain_model/dataset.py	22	25	88.0%

Totals
Change from base Build 4447217940:	86.0%
Covered Lines:	25615
Relevant Lines:	29788

---

💛 - Coveralls

[Panaetius]

should we separate renku.core.dataset.providers to make it clear what is for storage and what is for just add? It's getting a bit crowded in there.

And I'm a bit torn on the internal naming going from external to cloud, since it's not all cloud-based now. That could easily lead to misunderstandings. I like external or remote more

[m-alisafaee]

should we separate renku.core.dataset.providers to make it clear what is for storage and what is for just add? It's getting a bit crowded in there.

There are providers that are both storage and for add (S3, Azure, ...). I'm not sure how we can address them. I believe renku.core.dataset.providers isn't the best place for providers anymore. We should remove them outside the dataset module and put them directly under core. Maybe also rename them to data-provider (since we use provider in other places).

And I'm a bit torn on the internal naming going from external to cloud, since it's not all cloud-based now. That could easily lead to misunderstandings. I like external or remote more

I changed it to cloud since I've noticed people are using this term. Remote makes more sense although it's a bit overused. Let's discuss this in the meeting tomorrow to decide on the terminology.