v0.2.2

[v0.2.2] - 2025-04-08

Added

• inspect.py module with the following new functions:

  • show_shape – returns the shape of a DataFrame
  • show_dupes – counts duplicated rows
  • show_catvar – returns list of categorical (object/category) columns
  • show_convar – returns list of continuous (non-categorical) columns
  • show_lowcardvars – identifies categorical columns with low cardinality
  • show_binary_list – identifies binary columns (with/without NaNs)
  • count_rows_with_any_na – counts rows with at least one missing value
  • count_rows_with_all_na – counts rows where all values are missing
  • count_cols_with_any_na – counts columns with at least one missing value
  • count_cols_with_all_na – counts columns where all values are missing
  • count_total_na – returns the total number of missing values
  • count_unique_values – returns unique value counts per column
    (Docstrings included, generated by ChatGPT.)

• Unit tests for all eda_helpers.py functions.

• Test fixtures: create_na_test_df() and create_unique_test_df() in test_utils.py.

• test_s3.py unit tests for the aws/s3 module:

  • list_s3_contents
  • s3_file_to_dataframe

• Shared test fixtures in tests/conftest.py for mocking CSV/Excel downloads.

• Added automatic documentation via mkdocs

  • Add docs\ folder
  • mkdocs.yml
  • index.md and api.md

• eda/datetime.py module with datetime feature engineering functions:

  • create_dt_col() – creates a single derived datetime column (e.g., timestamp_year)
  • create_dt_cols() – supports creation of multiple datetime-derived columns in one call
    • Supports: "year", "month", "day", "weekday", "weekday_name", "weekofyear", "quarter", "dayofyear", "is_weekend", "is_month_start", "is_month_end"
    • Includes validation for unsupported types and missing columns
    • Raises error if inconsistent datetime formats (e.g., mixed "/" and "-") are detected

• Unit tests for eda/datetime.py:

  • Test coverage for both single and multi-column expansion
  • Error handling for:
    • Invalid column names
    • Unsupported col_types
    • Mixed datetime formats
    • Auto-conversion of string-formatted dates

• Refactored and consolidated test fixtures into tests/conftest.py:

  • Moved all fixtures from test_utils.py into conftest.py for automatic discovery across all test files
  • Includes fixtures: sample_df, na_test_df, unique_test_df, binary_list_df, datetime_df, dummy_csv_bytes, dummy_excel_bytes, mock_requests_get

• New test module: tests/unit/test_datetime.py

  • Organized tests specifically for datetime feature extraction utilities

• New dataio module with save_parquet() and load_parquet() functions for reliable Parquet I/O

• Type-safe and documented with NumPy-style docstrings

• Full test coverage using reusable sample_df fixture from conftest.py

• Unit test file test_io_utils.py to validate read/write functionality and nested directory creation

• save_csv() function to dataio module: cleanly saves DataFrames to CSV with default index=False and automatic directory creation

• load_csv() function to dataio module:

  • Attempts to load a CSV file using multiple common encodings (e.g. 'utf-8', 'latin1', 'cp1252')
  • Provides a helpful raw byte preview if all decoding attempts fail
  • Raises informative ValueError for failed loads to assist with debugging corrupted or misencoded files

• Unit tests for both save_csv() and load_csv() using sample_df and tmp_path

  • Includes fallback test simulating corrupted file behavior

• read_s3() to dataio.s3_io: loads public S3-hosted CSV or Excel files into a DataFrame with built-in error handling

• Unit tests for read_s3():

  • CSV and Excel load success
  • Handles unsupported file types (ValueError)
  • Graceful fallback on request errors (returns None)

• test_s3_utils.py to test list_s3_bucket() from aws.s3_utils with mocked boto3.client

Changed

• eda/__init__.py updated to expose all eda_helpers functions.

• list_unique_values() updated to handle both single column names and lists.

• test_utils.py simplified — now only contains helper functions if needed (no more @pytest.fixture)

• Updated test files (test_inspect.py, test_transform.py, etc.) to rely on auto-discovered fixtures via conftest.py

• eda/__init__.py updated to expose new datetime functions

• Renamed io.py to io_utils.py and moved to dataio/ to avoid namespace conflict with Python's built-in io module

• Updated all related imports to reflect new structure

• Renamed s3_file_to_dataframe() ➝ read_s3() for consistency with other I/O functions

• Split S3-related code:

  • list_s3_bucket() remains in aws.s3_utils
  • read_s3() moved to new dataio/s3_io.py

Fixed

• Corrected module import paths for internal packages (from jcds.aws.s3 import ...).

• Configured pytest.ini to include the project root in PYTHONPATH.

• Added input validation to create_dt_cols() to prevent datetime parsing issues with inconsistent formats

Notes

• Test discovery is working via pytest, and all tests are passing.
• Added moto and requests to environment.yml to support AWS and HTTP mocking.