0.14.0

• Implement table.add_files() method
• Fix issue where threads wouldn't exit while bulk downloading files from an index table
• General performance and reliability improvements when uploading data

0.13.0

What's Changed

• refactor: general cleanup and bugfixes by @imathews in #36

Full Changelog: v0.12.6...v0.13.0

0.12.6

• fix(datasets): fix bad reference in dataset returned from create_next_vesion (closes #32)
• feat(projects): add project.get(), project.exists(), and user.list_projects() methods

0.12.5

• Fix file path resolution on windows (#34, thanks @jbiggsets!)
• Fix an issue with the dataset.update() method
• Added support for REDIVIS_TMPDIR environment variable, which can be used to specify the temp directory used by this library (if not specified, the system default is used)

0.12.1

• fix table.list_files() error

0.12.0

This is a major update to the redivis-python client library, complete with significant performance improvements, new mechanisms for working with larger tables, and new integrations with pyarrow, polars, and dask. It also includes updates that allow for the generation of output tables within a redivis notebook environment.

• redivis-python library documentation
• Python notebooks on Redivis

New features 🚀

• New methods on Table, Query, and Upload classes:
  • to_dask_dataframe(): read a redivis table as a dask dataframe
  • to_polars_lazyframe(): read a redivis table as a polars lazyframe
  • to_arrow_table(): read a redivis table as a pyarrow table
  • to_arrow_dataset(): read a redivis table as a pyarrow dataset
  • to_arrow_batch_iterator(): iterate over a table via pyarrow RecordBatches
  • to_geopandas_dataframe(): read a redivis table with a geography variable as a geopandas dataframe
  • to_pandas_dataframe(): read a redivis table as a pandas dataframe for a table with a geography variable, using the new pyarrow dtypes by default (supercedes the old to_dataframe() method, which has been deprecated)
• Added batch_preprocessor argument to all above functions to efficiently pre-filter data as it is loaded
• Added dtype_backend argument to to_[geo]pandas_dataframe() methods. Specify either pyarrow, numpy, or numpy_nullable
• Add date_as_object boolean argument to_[geo]pandas_dataframe()` methods. Configure how dates should be stored in when using the legacy numpy dtypes.
• Add redivis.current_notebook(), returning an instance of the current notebook within a Redivis notebook environment (returns null in other environments)
  • Add Notebook.create_output_table(df) method to generate an output table from a redivis notebook

Performance ⚡

• Multi-threaded data streaming, alongside performance improvements in the underlying transfer protocol, should lead to at least a 3x improvement in data transfer speeds
• Reduced memory overhead significantly, with new mechanisms to load larger-than-memory data

Deprecations

• The to_dataframe method has been deprecated, in favor of the more explicit to_pandas_dataframe() and to_geopandas_dataframe() methods. The new methods default to the "pyarrow" dtype_backend — to continue using numpy as the pandas backend, you must specify dtype_backend="numpy" for the new methods. However, pyarrow is significantly faster, and supports nullable data types. It is recommended if numpy isn't required, and will generally work with existing code.

Dependencies

Bumped the following dependencies:

• pandas >= 2.0.0
• pyarrow >= 13.0.0
• pyarrow >= 0.14.0

Added the following dependencies:

• dask >= 2023.9.3
• polars >= 0.19
• shapely >= 2.0.1

0.11.1

• Fixed an issue with uploading large files with unicode encodings and/or non-text characters.
• Minor refactors to remove deprecated APIs

Note: v0.11.0 was skipped due to a mistake in the deployment process

0.10.0

Added support for external file transfers via a new transfer_specification parameter in upload.create() (documentation). This supports the use case of wanting to programmatically transfer files that reside in another environments, such as GCS, s3, a URL, BigQuery, or as another table on Redivis.

0.9.1

Fixed a bug where the upload of very large strings wasn't handled properly, causing 413 Request Entity Too Large errors. Strings should now be handled the same as other data inputs to uploads.

0.9.0

• All list_rows() methods now return a list of instances of the custom Row class, rather than named tuples. This works around an issue with named tuples in that their keys cannot start with an underscore character (e.g., _var1), partially resolving #29
  • In practice, the behavior and usage should be the same as before. You can access values in each row by position (index) -> row[0], by key like a dict -> row['_var1'], or as properties -> row._var1