Skip to content

Conversation

@cc-a
Copy link
Collaborator

@cc-a cc-a commented Jun 27, 2024

Description

A first mocked up version of what a revised data input layer may look like using standalone (i.e. no sqlalchemy) duckdb. So far implemented:

  • reader functions for the region, commodity definition and commodity demand tables that populate the their data into duckdb whilst also returning a numpy array representation.
  • calculate_global_commodities function that takes the data from the commodity definition table and transforms it into the xarray representation required by Muse internally.
  • tests for each function implemented plus additional tests addressing any constraints applied to database columns.

Mostly pushing this for visibility at this point as @tsmbland is planning to expand this to cover the remaining database tables and xarray data structures.

Fixes # (issue)

Type of change

Please add a line in the relevant section of
CHANGELOG.md to
document the change (include PR #) - note reverse order of PR #s.

  • New feature (non-breaking change which adds functionality)
  • Optimization (non-breaking, back-end change that speeds up the code)
  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (whatever its nature)

Key checklist

  • All tests pass: $ python -m pytest
  • The documentation builds and looks OK: $ python -m sphinx -b html docs docs/build

Further checks

  • Code is commented, particularly in hard-to-understand areas
  • Tests added that prove fix is effective or that feature works

@cc-a cc-a requested review from alexdewar and tsmbland June 27, 2024 16:32
@codecov
Copy link

codecov bot commented Jun 27, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.30%. Comparing base (f5bec10) to head (ae61de8).

Additional details and impacted files
@@               Coverage Diff               @@
##           new-data-input     #379   +/-   ##
===============================================
  Coverage           71.30%   71.30%           
===============================================
  Files                  44       44           
  Lines                5890     5890           
  Branches             1155     1155           
===============================================
  Hits                 4200     4200           
  Misses               1369     1369           
  Partials              321      321           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tsmbland tsmbland linked an issue Jun 28, 2024 that may be closed by this pull request
@cc-a cc-a mentioned this pull request Jun 28, 2024
8 tasks
@alexdewar alexdewar removed their request for review January 30, 2025 15:48
@tsmbland tsmbland closed this Aug 8, 2025
@github-project-automation github-project-automation bot moved this to ✅ Done in MUSE Aug 8, 2025
@tsmbland tsmbland deleted the duckdb branch August 8, 2025 11:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Create minimal example database with SQLAlchemy and/or DuckDB

3 participants