Implement unit-test for full datasets

[JoseEspinosa]

Is your feature request related to a problem?

Some of the module features can not be tested using the self-contained data-set of the repository.

Describe the solution you'd like

Implement unit testing that can fetch real datasets and this way could test that modules work with real data and that features such as default resource allocations etc. work as expected.

Describe alternatives you've considered

@KevinMenden, @maxulysse @drpatelh and myself(@JoseEspinosa) have discussed several ideas on Slack (see additional context) such as:

• Create a new repository to host the full datasets (or the pointers to them) similar to the test-datasets for modules
• Create a branch on the test-datasets that contains the links to the full modules data-sets.

However how to implement the tests and even how the data should be host still needs a lot of discussions and that is why we opened the issue, so feel free to contribute with your ideas 😃

Additional context

This issue has been opened following a discussion on Slack about how to organize test datasets and that the idea that at some point we may want to test the modules on real data.

[luisas]

@JoseEspinosa This is now done right? with the full test run on aws which is available now

[JoseEspinosa]

AFAIK this is not implemented. Although, I am not sure if there are plans to move this forward and how feasible it would be.

[luisas]

Ah ups, i thought it was solved by the full test on AWS, but it is at the pipeline level, sorry :)