Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The ability to specify a schema when using dfr open and dfr into-df #11634

Merged
merged 23 commits into from
Jan 29, 2024
Merged

The ability to specify a schema when using dfr open and dfr into-df #11634

merged 23 commits into from
Jan 29, 2024

Commits on Dec 22, 2023

  1. Initial work for specifying schemas: 

    Includes:
- Some type conversion from Value to polars schema
- Command for displaying the schema.
    @ayax79
    ayax79 committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    441834f View commit details
    Browse the repository at this point in the history

  2. Adding the ability to specify the schema when opening a csv.

    @ayax79
    ayax79 committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    4167d63 View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2024

  1. Implementing more string to schema dtype, formatting, errors

    @ayax79
    ayax79 committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    1a58ba9 View commit details
    Browse the repository at this point in the history

  2. trimming substrings for dtype

    @ayax79
    ayax79 committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    c1feb81 View commit details
    Browse the repository at this point in the history

  3. Allowing other types like json and jsonl to use --schema

    @ayax79
    ayax79 committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    95520d4 View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2024

  1. Adding command to show data types

    @ayax79
    ayax79 committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    056e748 View commit details
    Browse the repository at this point in the history

  2. Remove datatypes from usage

    @ayax79
    ayax79 committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    64ebec8 View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2024

  1. Removing InputType do use polars DataType directly

    @ayax79
    ayax79 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    696a605 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' into dataframe_schema

    @ayax79
    ayax79 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    c63862c View commit details
    Browse the repository at this point in the history

  3. Fixing errors due to changes in Value::Record

    @ayax79
    ayax79 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    fcf87e1 View commit details
    Browse the repository at this point in the history

  4. Added the ability to apply a schema when creating a dataframe

    @ayax79
    ayax79 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    c8bb803 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2024

  1. Make error message actually show the data type failure

    @ayax79
    ayax79 committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    723436e View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. Better handling of records and structs when specifying a schema.

    @ayax79
    ayax79 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    7ee0b03 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' into dataframe_schema

    @ayax79
    ayax79 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    394c1ae View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2024

  1. Added an example for specifying a schema to dfr into-df

    @ayax79
    ayax79 committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    6e6ad75 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' into dataframe_schema

    @ayax79
    ayax79 committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    809e612 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2024

  1. Merge branch 'main' into dataframe_schema

    @ayax79
    ayax79 committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    f6da7f7 View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2024

  1. fixing broken example test

    @ayax79
    ayax79 committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    1dd5f56 View commit details
    Browse the repository at this point in the history

  2. Change schema type and parameter specification to use <> instead of []

    @ayax79
    ayax79 committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    72857f3 View commit details
    Browse the repository at this point in the history

  3. dfr schema and dfr dtypes displaying with <>

    @ayax79
    ayax79 committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    32766eb View commit details
    Browse the repository at this point in the history

  4. preserve original behavior of dtypes

    @ayax79
    ayax79 committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    711834c View commit details
    Browse the repository at this point in the history

  5. dfr schema now returns structs appropriately

    @ayax79
    ayax79 committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    a37c410 View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2024

  1. fixing clippy errors

    @ayax79
    ayax79 committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    82da1d4 View commit details
    Browse the repository at this point in the history