Evaluate Cumulus as a Candidate for the Nucleus Trade Study

[ramesh-maddegoda]

It was decided to evaluate Cumulus as a Candidate for the Nucleus Trade Study.

Cumulus is a native cloud-based data ingest, archive, distribution, and management system used in Earth Observing System Data and Information System (EOSDIS) data streams.

Cumulus documentation is available at: https://nasa.github.io/cumulus/docs/cumulus-docs-readme

[tloubrieu-jpl]

@ramesh-maddegoda completed the trade study and asked questions to Mike Gangle to complete that.

[ramesh-maddegoda]

The trade study was updated at https://docs.google.com/spreadsheets/d/17Yxf1m6f4VvLd6QFuqJADAhwKPhqQnQemtMb9MqWyDI/edit#gid=2008405426 with Cumulus and the following list if questions were sent to Michael Gangl.

ETL Modes

1. Can Cumulus execute ETL jobs in batch mode where data is extracted, transformed and loaded as batches?
2. Can Cumulus execute ETL processes in real-time mode where data is extracted, transformed and loaded in real-time as soon as data is received (this can also be considered as streaming ETL)?

Pipeline Execution Control

3. Can Cumulus deactivate a data pipeline (so it will not get executed in future)?
4. Can Cumulus activate a deactivated data pipeline?
5. Can Cumulus suspend a data pipeline (in runtime)?
6. Can Cumulus resume a suspended data pipeline (in runtime)??
7. Can Cumulus terminate a data pipeline (in runtime)?

Pipeline Triggers

8. Can Cumulus trigger the execution of a data pipeline automatically based on a data file receive event (preferably on an S3 bucket?
9. Can Cumulus trigger the execution of a data pipeline automatically based on a change in a database table (Change Data Capture)?

Interfaces

10. Does Cumulus provide a Representative State Transfer (REST) based Application Programming Interface (API) to import and delete predefined data pipelines?
11. Does Cumulus provide a Command Line Interface (CLI) based Application Programming Interface (API) to import and delete predefined data pipelines?
12. Does Cumulus provide a python language wrapper for data pipeline monitoring (with ability to check progress and statistics on both current and historical pipelines)?

Security

13. Can Cumulus log each user access attempt to the system with the details of date and time of the event, outcome of the access attempt (allowed/denied), user name and user roles?
14. Does Cumulus provide role-based permissions to restrict data pipeline authoring, data pipeline management and data pipeline execution? If so, are these restrictions applicable to the operations invoked through other interfaces such as RESTful API and CLI?
15. Are data passes through Cumulus only accessible by authorized users/services?

Cost Optimization

16. Can Cumulus execute data pipelines on cost optimized infrastructure wherever possible?
17. Can Cumulus execute data pipelines in cost optimized timeframes whenever possible?

User Community

18. Does Cumulus have an external user community outside JPL and NASA?

[tloubrieu-jpl]

Waiting for answers from Mike Gangle

[ramesh-maddegoda]

Had a meeting with Michael Gangl and got answers to the question we had on Cumulus. Based on that the following Nucleus trade study was updated with Cumulus.
https://docs.google.com/spreadsheets/d/17Yxf1m6f4VvLd6QFuqJADAhwKPhqQnQemtMb9MqWyDI/edit#gid=1611047106