Create generic dataset_base and s3_dataset modules from current datasets

[dlpzx]

Is your feature request related to a problem? Please describe.
It is difficult to add new dataset types with the current implementation of the datasets module.

Describe the solution you'd like
If we want to add more types of dataset (e.g. Redshift dataset) a first step will be the creation of a generic class that defines a data.all Dataset abstraction that can be used by each of the particular dataset implementations.

Describe alternatives you've considered
This are rough design considerations that might change during implementation:

---

Backend

1. Avoid circular dependencies and remove the current datasets_base - [COMPLETED ✅ ]

There is some work to do to avoid circular dependencies between the initial datasets and dataset_sharing. There are some API calls that use each others' graphql models. From a code logic perspective, datasets should not depend on dataset_sharing code, so as a first step we will remove this dependency by defining the corresponding code related to dataset_sharing in dataset_sharing always.

2. Define basic code layout

• datasets_base --> includes db base models for Dataset + list APIs that are generic
  - DEPENDS ON: nothing - we need to remove dependencies with sharing. Anything related to sharing should be moved to the sharing module.
• dataset_sharing_base --> includes db base models for ShareObject, base permissions + list methods
  - DEPENDS ON: datasets_base
• s3_datasets --> includes all db models of S3 datasets + implementation S3 APIs
  - DEPENDS ON: datasets_base and dataset_sharing_base (maybe)
• s3_dataset_sharing ---> business logic for S3 sharing
  - DEPENDS ON: s3_dataset, dataset_sharing_base
• in the future redshift_datasets and redshift_dataset_sharing

3. datasets_base module

1. contains the API calls that list Datasets that are generic (without details to S3/Glue)
2. contains the base code for other modules to implement a dataset

I thought of splitting the module into 2 modules: datasets_list module and datasets_base module. Although datasets_base would not be a module, it is some base code that will be re-used by other modules. Because of this and because the dataset_list uses dataset_base models I decided to leave it in the same module, but we could change this behavior, it is not a one way door.

Base code for Datasets

• NO API CALLS
• Base service dataset_base_service - Defines abstract class DatasetBaseService to be implemented in each dataset type --> DISCARDED: The reason why we wanted to implement this interface was to ensure that the dataset methods were decorated with the CREATE_DATASET... permissions. But as explained below, it does not make much sense to aply the same permissions to the different types of datasets, except for the list permissions which would not be par tof this base class.
• Database models - defined in datasets/db/dataset_models
  - DatasetBase - same as before but WITHOUT all S3-related logic. We add the datasetType column. Each dataset module will have to implement their own tables based on this table. For this we can use sqlalchemy *joined table inheritance
  - DatasetLock - it is a generic model that can be used by any type of dataset.
• Database operations - in DatasetRepository in datasets/db/dataset_repositories DatasetLock transactions. Each dataset module will have to implement their own repositories.
• datasets_enums - re-used by all Datasets

*Joined table inheritance: In joined table inheritance, each class along a hierarchy of classes is represented by a distinct table. Querying for a particular subclass in the hierarchy will render as a SQL JOIN along all tables in its inheritance path. If the queried class is the base class, the base table is queried instead, with options to include other tables at the same time or to allow attributes specific to sub-tables to load later.

This means that if we query the table datasets, base class, we get a list of all the generic attributes of all types of datasets. If we query the specific subclass s3_datasets it retrieves the generic details+the S3 details of the S3 datasets.

List Datasets

• API definitions in datasets/api
• Service dataset_list_service - business logic and permissions checking
• Permissions dataset_base_permissions DATASET PERMISSIONS FOR ENVIRONMENT - List Datasets in environment needed for the service
• Database models - explained above
• Database operations to query all Datasets DatasetListRepository in datasets/db/dataset_repositories independent from their dataset type. ~~It is a EnvironmentResource type to count_resources before deleting an environment. ~~ DISCARDED: it makes more sense to count the specific resources before deletion.
• datasets_enums - explained above

4. s3_datasets module

Very similar to the previous datasets_module, it contains the code to manage S3-GLUE datasets in data.all.

• API calls - WITHOUT generic listDataset calls
• AWS clients - includes all previous code as it is S3/GLUE-Dataset specific
• CDK stacks and permissions
• Database models/repositories:
  - S3Dataset model that uses Dataset as base using inheritance in sqlalchemy.
  - Other models that are specific to S3/GLUE-Datase: tables, folders.....
• handlers - includes all previous code as it is S3/GLUE-Dataset specific
• indexers
• services
• tasks - includes all previous code as it is S3/GLUE-Dataset specific

Frontend

for s3_datasets and datasets_base

• The listDatasets view is generic as it is the API listDatasets and analogous apis
• All the detail getDataset views are specific of each Dataset type

Config.json

for s3_datasets and datasets_base

Divide the parameters between those that are generic dataset parameters (defined in dataset module) and those that are s3_datasets specific parameters.

        "datasets": {
            "active": true,
            "features": {
                "share_notifications": {
                    "email": {
                        "active": false,
                        "parameters": {
                            "group_notifications": true
                        }
                    }
                },
                "confidentiality_dropdown" : true,
                "topics_dropdown" : true
            }
        },
        "s3_datasets": {
            "active": true,
            "features": {
                "file_uploads": true,
                "file_actions": true,
                "aws_actions": true,
                "preview_data": true,
                "glue_crawler": true
            }
        },

---

Related to #955

🥇 Thanks to @dosiennik for his suggestion of using inheritance in sqlalchemy: https://docs.sqlalchemy.org/en/20/orm/inheritance.html this will allow us to define a Dataset model and child models

[dlpzx]

Step-by-step implementation plan

Pre-requisites

• decouple datasets and datasets_sharing and remove the initial datasets_base module

1. Split datasets

Separate datasets into datasets_base and s3_datasets
We need to implement it in multiple PRs, otherwise it will be impossible to follow.
*BE=Backend, FE=Frontend

• (BE + FE) Rename datasets to s3_datasets (PART1 PR)
• (BE ONLY) Create datasets_base module and update the dependencies (s3_datasets depends on datasets_base) (PART2)
• (BE ONLY) Move generic dataset enums to datasets_base (PART2)
• (BE ONLY) Create a base DatasetBase model in datasets_base and an S3Dataset model in s3_dataset that inherits it. (use enum DatasetType) Ensure deletes deletes items in both tables and create migration scripts. (PART3)
• (BE ONLY) Move generic db models and generic services to datasets_base = DatasetLock(PART4)
• (BE+FE) Create list APIs in datasets_base, adapt the frontend and move the generic DatasetInterface to datasets_base (PART5) and additional PR
• (FE ONLY) Create new Datasets_Base FE module and add generic views and services (PART6)
• (BE ONLY) Review dataset permissions and if it makes sense split permissions into dataset_base permissions and S3_dataset permissions (PART7) ---> ⚠️ DISCARDED. LIST_ENVIRONMENT_DATASETS is the only permission that should be in datasets_base, the rest are specific for each type of dataset. For example one group might have permissions to CREATE_S3_DATASETS but not to CREATE_REDSHIFT_DATASETS. They are protecting different API calls, therefore they should be different permissions. The only change that would be nice is renaming the current CREATE_DATASETS permissions to CREATE_S3_DATASETS permissions. It is a cosmetic change that would be nice, but it involves some effort and carefully leading with the alembic migration. For the moment I have not implemented this change.
• (BE ONLY) Review CatalogIndexer and if it makes sense add it to datasets_base (PART8)---> ⚠️ DISCARDED The DatasetCatalogIndexer of S3 datasets upserts tables and folders that are particular to s3 datasets. Despite involving some duplication of code it is good that each dataset module has control over the catalog indexing.

2. Split config.json

Separate config.json parameters into dataset_base parameters and s3_datasets parameters

• BE+FE - split the config.json params and the references in BE and FE (PART7)

3. Split dataset sharing

Separate datasets_sharing into sharing_base and s3_datasets_sharing
in separate issue: #1283

[dlpzx]

Completed!