Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read index API #1568

Open
wants to merge 48 commits into
base: master
Choose a base branch
from
Open

Read index API #1568

wants to merge 48 commits into from

Commits on May 31, 2024

  1. Add restriction on the version of grcp needed to install ArcticDB

    New versions require protobuf >= 5 which we don't support
    Vasil Pashov committed May 31, 2024
    Configuration menu
    Copy the full SHA
    136721b View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. Merge branch 'master' of github.com:man-group/ArcticDB

    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    981bdae View commit details
    Browse the repository at this point in the history
  2. Add TDD style tests for the new read_index API

    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    2f9a64c View commit details
    Browse the repository at this point in the history
  3. Expand tests to cover more ground

    * Different versions of as_of
    * date_rage
    * row_range
    * Different column slice size
    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    fd13b8e View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c974cdc View commit details
    Browse the repository at this point in the history
  5. Add entry point for read_index_columns in python

    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    d941d20 View commit details
    Browse the repository at this point in the history
  6. Add LocalVersionedEngine::get_version_identifier

    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    bf23086 View commit details
    Browse the repository at this point in the history
  7. Add read_index_columns_internal funciton and call it from the public …

    …interface
    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    bd6e881 View commit details
    Browse the repository at this point in the history
  8. Separate build read_query_filters into two functions one for rows and…

    … one for cols
    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    5ac734e View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    264f0ff View commit details
    Browse the repository at this point in the history
  10. Pass the pipeline as a const bare pointer when setting the selected c…

    …olumns
    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    254a49a View commit details
    Browse the repository at this point in the history
  11. Comment bucketize_dynamic in the proto file

    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    94817d8 View commit details
    Browse the repository at this point in the history
  12. Remove unused function

    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    f1983f2 View commit details
    Browse the repository at this point in the history
  13. Refactor

    * Use shared_ptr for the pipeline context. It's needed because when
      queries are created a lambda captures the pointer and then the lambda
      object is returned. We want it to be copied so that the lambda is an
      owner.
    * Add comments to pipeline fields (selected_columns_ and
      overal_column_bitset_)
    * Fix pipeline context swap as it did not copy the overal column bitset
    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    7b86694 View commit details
    Browse the repository at this point in the history
  14. Read index columns in the CPP layer

    Mark the correct columns both in the selected_columns and overall_column_bitset.
    After that the dataframe is read as usual and returned. The python layer
    needs additional work since it cannot denormalize it (probably because
    there is only one index column with no data columns).
    
    TODO:
    * The C++ is probably doing more than needed
    * Python denomalization
    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    16e87f1 View commit details
    Browse the repository at this point in the history
  15. Fix read index for when range index is used

    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    45102ca View commit details
    Browse the repository at this point in the history
  16. Use range based for in modify_descriptor

    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    fc41710 View commit details
    Browse the repository at this point in the history
  17. Improve the way index columns are selected

    The index columns are always the first columns in the stream descriptor.
    Thus we can check the count of the index columns and add the first n
    columns to the selected set, skipping the procedures of creating hash
    sets and checking if a field name is in the set.
    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    0b287ed View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    4e3c031 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    8aaf576 View commit details
    Browse the repository at this point in the history
  20. Fix compilation error

    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    f30a5b0 View commit details
    Browse the repository at this point in the history
  21. Merge branch 'master' of github.com:man-group/ArcticDB

    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    162c199 View commit details
    Browse the repository at this point in the history
  22. Fix bug introduced with earlier commit

    Vasil Pashov committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    951cee7 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. Denormalize multiindex

    Vasil Pashov committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    a132759 View commit details
    Browse the repository at this point in the history
  2. Add comments and change function declaration

    Vasil Pashov committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    3ffdc43 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. Configuration menu
    Copy the full SHA
    19984b2 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. Fix the way slices and keys are selected for a dataframe without row …

    …filter
    Vasil Pashov committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    df6549b View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Enable more tests and fix some bugs in the tests

    Vasil Pashov committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    05b63dc View commit details
    Browse the repository at this point in the history
  2. Fix fetching of empty indexes

    Vasil Pashov committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    6d50233 View commit details
    Browse the repository at this point in the history
  3. Set rowcount after reading rowrange index. Add python binding to get …

    …the rowcount
    Vasil Pashov committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    7466696 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. Extract _compute_filter_start_end_row into a function

    Vasil Pashov committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    13c57ab View commit details
    Browse the repository at this point in the history
  2. Throw exception in case of multi key, pickled data, custom normalizer…

    …s or recursive normalizers when trying to read index
    Vasil Pashov committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    fbd6651 View commit details
    Browse the repository at this point in the history
  3. Add tests for normalization, pickled data and staged writes for read …

    …index functionality
    Vasil Pashov committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    f6ef6ee View commit details
    Browse the repository at this point in the history
  4. Add index fixture

    Vasil Pashov committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    164138d View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Refactor things in read_index_columns_impl

    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    a82df01 View commit details
    Browse the repository at this point in the history
  2. Remove test cases which don't make sense at the moment

    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    17f1915 View commit details
    Browse the repository at this point in the history
  3. Fix CI compilation errors

    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    2f9ff35 View commit details
    Browse the repository at this point in the history
  4. Merge branch 'master' of github.com:man-group/ArcticDB

    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    074be79 View commit details
    Browse the repository at this point in the history
  5. Merge branch 'master' into read-index-merge

    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    fa753f2 View commit details
    Browse the repository at this point in the history
  6. Fix setup.cfg

    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    36d0c7a View commit details
    Browse the repository at this point in the history
  7. Fix compilation errors in the tests

    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    9f42bf2 View commit details
    Browse the repository at this point in the history
  8. Fix pytest fixture for read_index_columns

    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    4a8b596 View commit details
    Browse the repository at this point in the history
  9. Fix failing test

    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    d79d882 View commit details
    Browse the repository at this point in the history
  10. Change fixture scopes

    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    bd31991 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    29518c8 View commit details
    Browse the repository at this point in the history
  12. Test differences between pandas 1 and pandas 2 when reading index col…

    …umns
    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    43c535a View commit details
    Browse the repository at this point in the history
  13. Omit library name from lmdb_version_store_row_slice

    Vasil Pashov committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    3eb997a View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    a626b4d View commit details
    Browse the repository at this point in the history