Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-43079: Add metadata module for encapsulating sqlalchemy binding #47

Merged
merged 37 commits into from
Apr 1, 2024

Commits on Mar 13, 2024

  1. Set autoincrement field to None by default

    The current logic in Felis's sql module assumes that autoincrement
    will be None if it is not set explicitly so for now match this logic
    in the datamodel.
    JeremyMcCormick committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    08aa636 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2024

  1. Add metadata module and implementation of SchemaMetaData class

    Add a module with a new class that encapsulates the creation of
    sqlalchemy MetaData from a Pydantic Schema object, along with a test
    that dumps the corresponding SQL DDL statements to a file. The logic
    for translating into SQA was primarily derived from the existing
    visitor class in sql.py, which this is intended to replace eventually.
    JeremyMcCormick committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    bc05ec4 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. Add command line interface that uses metadata module

    Add a "create" command that uses the metadata module instead of the
    sql module.
    JeremyMcCormick committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    0eba714 View commit details
    Browse the repository at this point in the history
  2. Add schema name and create if not exists cli flags

    Add flags for specifying an alternate schema name and creating the
    schema in the database if it does exist already. For MySQL, a new
    database will be created. In PostgreSQL, a new schema will be created
    within the current database.
    JeremyMcCormick committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    8ebd4f8 View commit details
    Browse the repository at this point in the history
  3. Add ability to drop and create schema using the command line interface

    Add several new flags to the CLI for dropping a schema if it exists and
    creating one if it does not exist.
    
    Add methods implementing this functionality to the metadata module.
    JeremyMcCormick committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    64d23a8 View commit details
    Browse the repository at this point in the history
  4. Add simple YAML schema for testing

    This is a generic schema representing sales data with customers and
    orders for testing purposes.
    JeremyMcCormick committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    460c805 View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2024

  1. Configuration menu
    Copy the full SHA
    6e885ee View commit details
    Browse the repository at this point in the history
  2. Add test case for creating metadata from a schema and loading into a db

    Add a test case which loads a simple YAML test file into a schema,
    creates SQLAlchemy Metadata from the object, creates the database
    objects within an in-memory SQLite database, and then checks that the
    reflected MetaData is the same as that created from the schema.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    ca74c58 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ff56648 View commit details
    Browse the repository at this point in the history
  4. Comment out bad value in test data

    This value is not handled correctly as it ends up being quoted. It will
    be fixed on another branch.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    fe5b126 View commit details
    Browse the repository at this point in the history
  5. Add wrapper class for executing SQL on an engine or mock connection

    This is used by the metadata module to handle a SQA engine or mock
    connection. SQL is executed in a with statement for an engine but not
    for a mock connection, as it does not support this.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    e2fb750 View commit details
    Browse the repository at this point in the history
  6. Move sql variant utility functions into a new module

    The sql module is being deleted but the variant handling utilities
    need to be kept.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    dd95926 View commit details
    Browse the repository at this point in the history
  7. Remove sql module and its tests

    Remove the sql module which is replaced by the metadata module.
    
    Remove the "create-all" command from the CLI which is replaced by
    "create".
    
    Remove the InsertDump class from the cli module, as it was moved into
    the metadata module.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    63070af View commit details
    Browse the repository at this point in the history
  8. Add builder class for creating SQLAlchemy MetaData from a Schema

    Instead of having a class with internal functions for building a
    MetaData object, use a builder pattern for a cleaner interface.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    921831d View commit details
    Browse the repository at this point in the history
  9. Correct cli handling for sqlite and implement automatic dry run

    Instead of not setting the schema name, use "main" instead which works
    for sqlite. In case a host name is not set on the database engine URL,
    force a dry run using the supplied variant.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    e61dde1 View commit details
    Browse the repository at this point in the history
  10. Add tests for indexes and constraints in MetaData test case

    Test that the metadata from the builder and that reflected from the
    database have equivalent index and constraint objects. Add some
    additional constraints to the YAML test file.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    b0a01f1 View commit details
    Browse the repository at this point in the history
  11. Add flags for applying the schema name to the metadata and tables

    In some cases, it is not desirable to apply the schema name to either
    the metadata or the tables. Add two flags to the builder for
    controlling this behavior. The flags are true by default and can be
    set to false if the user does not want the schema to be applied.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    5ce2533 View commit details
    Browse the repository at this point in the history
  12. Prepend underscore to private global variable and change regexp

    Change the regular expression to match on numbers rather than any
    character between the parentheses.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    dd12e4d View commit details
    Browse the repository at this point in the history
  13. Use IO type so name attribute can be directly accessed

    Fix mypy issue where TextIOBase does not have a visible name attribute.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    0d28a8a View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    bc5c4d2 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    9f8a844 View commit details
    Browse the repository at this point in the history
  16. Add utility method for getting column's datatype with variants

    Provide a utility method for getting the SQLAlchemy datatype from a
    column object in the Pydantic model, including the variant type
    handling.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    88d5efe View commit details
    Browse the repository at this point in the history
  17. Add test comparing metadata to the schema from which it was built

    This test reads in a test schema, creates a SQLAlchemy MetaData object
    from it, and then compares the schema to the metadata to check that
    they have the equivalent, expected information.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    05fd371 View commit details
    Browse the repository at this point in the history
  18. Simplify handling of different constraint types in builder method

    Remove exception checking for each constraint type as it is
    unnecessary. In the unlikely case that a constraint was not built
    properly, throw an exception.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    238ce62 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    01ecdc2 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    c922293 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    eaa3424 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    ab96c6d View commit details
    Browse the repository at this point in the history
  23. Add 'postgresql:datatype' field to Column

    This field is required so that a PostgreSQL datatype can be optionally
    provided in the column information.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    e9afb14 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    410aa47 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    02bfc75 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    73c5fb5 View commit details
    Browse the repository at this point in the history
  27. Remove use_enum_values from Pydantic data model

    This setting confuses mypy and the syntax seems clearer when using the
    enum object instead.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    3cd31a2 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    9eb8e95 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    1f6a0a4 View commit details
    Browse the repository at this point in the history
  30. Change primaryKey to primary_key in Pydantic model for consistency

    All of the other fields use snake case and aliases for camelcase
    variables, so change primary key to match.
    JeremyMcCormick committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    35a6c42 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. Configuration menu
    Copy the full SHA
    b955513 View commit details
    Browse the repository at this point in the history