Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

modularization: Datasets modularization pt.4 #441

Merged
merged 75 commits into from
May 4, 2023
Merged

modularization: Datasets modularization pt.4 #441

merged 75 commits into from
May 4, 2023

Commits on Apr 11, 2023

  1. Initialization of dataset module

    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    3a5e0de View commit details
    Browse the repository at this point in the history

  2. Refactoring of datasets 

    Moved dataset table column to modules
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    a50a02f View commit details
    Browse the repository at this point in the history

  3. Refactoring of datasets 

    Renamed table column to the python's convention format
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    be14986 View commit details
    Browse the repository at this point in the history

  4. Refactoring of datasets 

    Added dataset module to config.json
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    06f82ad View commit details
    Browse the repository at this point in the history

  5. Fixed leftover in loader

    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    38145ae View commit details
    Browse the repository at this point in the history

  6. Dataset refactoring 

    Moved database table service
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    f0e146a View commit details
    Browse the repository at this point in the history

  7. Dataset refactoring 

    Renamed DatasetTable to DatasetTableService to avoid collisions with models.DatasetTable
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    b039163 View commit details
    Browse the repository at this point in the history

  8. Dataset refactoring 

    Moved DatasetTableColumn to modules
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    b7922ed View commit details
    Browse the repository at this point in the history

  9. Notebooks doesn't require tasks

    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    1771bca View commit details
    Browse the repository at this point in the history

  10. Renamed tasks to handlers 

    Currently, only async handlers require dedicated loading. Long-running tasks (scheduled tasks) might not need to have a dedicated loading mode
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    3d1603f View commit details
    Browse the repository at this point in the history

  11. Dataset refactoring 

    Extracted code from glue to glue_column_handler
Added handlers importing for datasets
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    fb6b515 View commit details
    Browse the repository at this point in the history

  12. Dataset refactoring 

    Extracted the code for dataset table handler
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    e3596a5 View commit details
    Browse the repository at this point in the history

  13. Dataset refactoring 

    Extracted the long-running task for datasets
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    3af2ecf View commit details
    Browse the repository at this point in the history

  14. Dataset refactoring 

    Extracted the subscription service into datasets
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    1a063b2 View commit details
    Browse the repository at this point in the history

  15. Dataset refactoring 

    Extracted the handler to get table columns
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    b733714 View commit details
    Browse the repository at this point in the history

  16. Extracted feed registry 

    Needed for migration for modules
    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    2a4e2e0 View commit details
    Browse the repository at this point in the history

  17. Extracted feed and glossary registry and created a model registry

    @nikpodsh
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    c15d090 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2023

  1. Dataset refactoring 

    Fixed tests and added new for dataset module
    @nikpodsh
    nikpodsh committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    052a2b1 View commit details
    Browse the repository at this point in the history

  2. Fixed and unignored test_tables_sync

    @nikpodsh
    nikpodsh committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    d984483 View commit details
    Browse the repository at this point in the history

  3. Split model registry into feed and glossaries 

    Glossaries had different target types and had to be treated differently
    @nikpodsh
    nikpodsh committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    dc0c935 View commit details
    Browse the repository at this point in the history

  4. Abstraction for glossaries 

    Created API for glossaries to use modularization
    @nikpodsh
    nikpodsh committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    727e353 View commit details
    Browse the repository at this point in the history

  5. Fixed leftovers

    @nikpodsh
    nikpodsh committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    49fbb41 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2023

  1. Datasets refactoring 

    Added and fixed tests
    @nikpodsh
    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    7d029e7 View commit details
    Browse the repository at this point in the history

  2. Added runtime type registration for Union GraphQL type

    @nikpodsh
    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    be527eb View commit details
    Browse the repository at this point in the history

  3. Changed Feed type registration mechanism 

    Moved FeedRegistry to gql since it's more appropriate place for this
Started using registry to provide types
Renaming and small fixes
    @nikpodsh
    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    3daf2aa View commit details
    Browse the repository at this point in the history

  4. Added TODO for future refactoring 

    Solve circular dependecy for redshift. It should go away after the migration of redshift
    @nikpodsh
    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    db3bfd3 View commit details
    Browse the repository at this point in the history

  5. Added GlossaryRegistry for Union scheme

    @nikpodsh
    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    13b6e92 View commit details
    Browse the repository at this point in the history

  6. Changed import in redshift module

    @nikpodsh
    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    144dfea View commit details
    Browse the repository at this point in the history

  7. No need for Utils yet

    @nikpodsh
    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    d43b9b3 View commit details
    Browse the repository at this point in the history

  8. Fixed linting

    @nikpodsh
    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    39b244c View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2023

  1. Datasets refactoring 

    Moving datasets profiling to datasets modules
    @nikpodsh
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    cb3800a View commit details
    Browse the repository at this point in the history

  2. Datasets refactoring 

    Renaming profiling
    @nikpodsh
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    dd8e597 View commit details
    Browse the repository at this point in the history

  3. Datasets refactoring 

    Renaming table_column_model to models to easier import other models
    @nikpodsh
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    8ca7bea View commit details
    Browse the repository at this point in the history

  4. Datasets refactoring 

    Moving DatasetProfilingRun model
    @nikpodsh
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    e36ab3b View commit details
    Browse the repository at this point in the history

  5. Datasets refactoring 

    Moving dataset profiling service and renaming it
    @nikpodsh
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    31720c2 View commit details
    Browse the repository at this point in the history

  6. Datasets refactoring 

    Extracted glue_profiling_handler
    @nikpodsh
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    8a907df View commit details
    Browse the repository at this point in the history

  7. Datasets refactoring 

    Deleted DatasetTableProfilingJob since could not find any usage of it
    @nikpodsh
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    561da72 View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2023

  1. Datasets refactoring 

    Moved dataset storage location into modules
    @nikpodsh
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    73c8150 View commit details
    Browse the repository at this point in the history

  2. Datasets refactoring 

    Renamed dataset storage location
    @nikpodsh
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    56a3610 View commit details
    Browse the repository at this point in the history

  3. Datasets refactoring 

    Returned the name to model after renaming the service
    @nikpodsh
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    47a38cc View commit details
    Browse the repository at this point in the history

  4. Datasets refactoring 

    Moved DatasetStorageLocation into modules
    @nikpodsh
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    dbb5517 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'datasets-mod-part2' into datasets-mod-part3

    @nikpodsh
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    a3def13 View commit details
    Browse the repository at this point in the history

  6. Datasets refactoring 

    Moved DatasetStorageLocation into dataset services
    @nikpodsh
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    b256678 View commit details
    Browse the repository at this point in the history

  7. Datasets refactoring 

    Extracted s3_location_handler
    @nikpodsh
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    66b5ddb View commit details
    Browse the repository at this point in the history

  8. Datasets refactoring 

    Moved the dataset stack into modules
    @nikpodsh
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    352d824 View commit details
    Browse the repository at this point in the history

  9. Datasets refactoring 

    Moved indexing into GlossaryRegistry
    @nikpodsh
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    9934a9c View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2023

  1. Datasets refactoring 

    Removed dead code
    @nikpodsh
    nikpodsh committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    228c175 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2023

  1. Datasets refactoring 

    Extracted dataset share service
    @nikpodsh
    nikpodsh committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    263d10c View commit details
    Browse the repository at this point in the history

  2. Datasets refactoring 

    Solved broken reference
    @nikpodsh
    nikpodsh committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    417e6e5 View commit details
    Browse the repository at this point in the history

  3. Introduced Indexers

    @nikpodsh
    nikpodsh committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    7aaff5b View commit details
    Browse the repository at this point in the history

  4. Extracted upsert_dataset_folders into DatasetLocationIndexer and rena… 

    …med it
    @nikpodsh
    nikpodsh committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    4e31b99 View commit details
    Browse the repository at this point in the history

  5. Moved DatasetLocationIndexer into the dataset module

    @nikpodsh
    nikpodsh committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    b772812 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2023

  1. Moved DatasetStorageLocation methods to the service

    @nikpodsh
    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    cd798e2 View commit details
    Browse the repository at this point in the history

  2. Renamed the service

    @nikpodsh
    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    b0e6a62 View commit details
    Browse the repository at this point in the history

  3. Moved DatasetIndexer to modules

    @nikpodsh
    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    27c6d79 View commit details
    Browse the repository at this point in the history

  4. Created a dataset repository. 

    There is a few of circular imports. It's a first attempt to solve it
    @nikpodsh
    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    0e730ac View commit details
    Browse the repository at this point in the history

  5. Moved DatasetTableIndexer

    @nikpodsh
    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    9ac7964 View commit details
    Browse the repository at this point in the history

  6. Fixed test mocking

    @nikpodsh
    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    a1825ba View commit details
    Browse the repository at this point in the history

  7. Fixed circular import while half of the module is not migrate

    @nikpodsh
    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    d295485 View commit details
    Browse the repository at this point in the history

  8. Removed not used alarms

    @nikpodsh
    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    005a5e7 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2023

  1. Moved dataset table GraphQL api in modules

    @nikpodsh
    nikpodsh committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    0fd7c02 View commit details
    Browse the repository at this point in the history

  2. Moved DatasetTable model to modules

    @nikpodsh
    nikpodsh committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    7030c82 View commit details
    Browse the repository at this point in the history

  3. Moved delete_doc to BaseIndexer

    @nikpodsh
    nikpodsh committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    ba45ca5 View commit details
    Browse the repository at this point in the history

  4. Lazy creation of connection to OpenSearch

    @nikpodsh
    nikpodsh committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    dc8ff72 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2023

  1. Resolved code conflict

    @nikpodsh
    nikpodsh committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    2ac3ae7 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2023

  1. Merge remote-tracking branch 'origin/datasets-mod-part2' into dataset… 

    …s-mod-part3
    @nikpodsh
    nikpodsh committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    fca218f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'datasets-mod-part3' into datasets-mod-part4 

    # Conflicts:
#	backend/dataall/modules/datasets/handlers/glue_column_handler.py
#	tests/api/test_dataset_table.py
    @nikpodsh
    nikpodsh committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    ef98aa0 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2023

  1. Merge remote-tracking branch 'upstream/modularization-main' into data… 

    …sets-mod-part3

# Conflicts:
#	backend/dataall/db/api/__init__.py
#	backend/dataall/modules/datasets/__init__.py
#	backend/dataall/modules/datasets/api/__init__.py
#	backend/dataall/modules/datasets/db/models.py
    @nikpodsh
    nikpodsh committed May 2, 2023
    Configuration menu
    Copy the full SHA
    2cd14e0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'datasets-mod-part3' into datasets-mod-part4 

    # Conflicts:
#	backend/dataall/modules/datasets/db/models.py
    @nikpodsh
    nikpodsh committed May 2, 2023
    Configuration menu
    Copy the full SHA
    a3d9676 View commit details
    Browse the repository at this point in the history

Commits on May 4, 2023

  1. Review remarks

    @nikpodsh
    nikpodsh committed May 4, 2023
    Configuration menu
    Copy the full SHA
    f382a68 View commit details
    Browse the repository at this point in the history

  2. Added TODO

    @nikpodsh
    nikpodsh committed May 4, 2023
    Configuration menu
    Copy the full SHA
    532ff0d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'datasets-mod-part3' into datasets-mod-part4

    @nikpodsh
    nikpodsh committed May 4, 2023
    Configuration menu
    Copy the full SHA
    afcae66 View commit details
    Browse the repository at this point in the history

  4. After the merge of part3

    @nikpodsh
    nikpodsh committed May 4, 2023
    Configuration menu
    Copy the full SHA
    61d4eb3 View commit details
    Browse the repository at this point in the history

  5. Fixed imports

    @nikpodsh
    nikpodsh committed May 4, 2023
    Configuration menu
    Copy the full SHA
    72864d6 View commit details
    Browse the repository at this point in the history

  6. Update tests/api/conftest.py 

    Co-authored-by: dbalintx <132444646+dbalintx@users.noreply.github.com>
    @nikpodsh @dbalintx
    nikpodsh and dbalintx committed May 4, 2023
    Configuration menu
    Copy the full SHA
    07e6975 View commit details
    Browse the repository at this point in the history