Include optimization scalar

[glatterf42]

Summary

This PR expands the data model required for message_ix by adding limited support for Scalars. One can now create, get, list, tabulate, and manage docs for scalars with the following syntax:

Implementation Details

# In the core layer:
test_mp = Platform(_backend=RestTestBackend(test_sqlite_mp.backend))
run = test_mp.runs.create("Model", "Scenario")
unit = test_mp.units.create("Test Unit")
# To create a new scalar:
# NOTE: the combination of (Run.id, Scalar.name) is checked for uniqueness
        scalar_1 = run.optimization.scalars.create(
            "Scalar 1", value=10, unit="Test Unit"
        )

# To list existing scalars:
scalars = run.optimization.scalars.list()

# To get a single scalar:
scalar = run.optimization.scalars.list(name="Scalar 1")
# or:
scalar = run.optimization.scalars.get("Scalar 1")

# To tabulate a scalar by specifying a name:
scalar_1 = run.optimization.scalars.tabulate(name="Scalar 1")

# To update a scalar:
        scalar = run.optimization.scalars.create(
            "Scalar", value=10, unit=unit.name
        )
        assert scalar.value == 10
        assert scalar.unit.id == unit.id

        scalar.value = 30
        scalar.unit = "Test Unit"

        result = run.optimization.scalars.get("Scalar")

        assert scalar.id == result.id
        assert scalar.name == result.name
        assert scalar.value == result.value == 30
        assert scalar.unit.id == result.unit.id == 1

# See tests/data/test_optimization_scalar.py to see the syntax for the data layer.

Further Changes

While working on this PR, I added/updated a few other minor things:

• Added tests for the data layer functionality of IndexSets
• Updated/expanded IndexSet tests
• Added a data-layer get_by_id function for Units

TO-DOs

• Add migrations once the DB model is accepted

[glatterf42]

Please note that there is support for upsert operations in sqlalchemy for sqlite and postgresql. I opted for separate create() and update() functionality because our current create() functions employ the session.add statement, which seems to be different from the INSERT statement that is required for upserts. We would probably also have to manage the different dialect-specific insert() functions in order to employ their insert().on_conflict_do_update() functionality, since this doesn't exist for all backends. Thus, keeping create() and update() separate seemed easier and more aligned with current infrastructure. If more update functionality is desired, it might however be beneficial to look into the existing upsert functions and maybe add an Upserter mixin.

[meksor]

What does "doesnt like lists" mean? assert_unordered_equality is meant only for DFs afaik...

[glatterf42]

It is, that's why it didn't work when I tried to pass in a list of DFs. That's why I added this auxiliary df_from_list() function. We could rephrase this hint as "... only accepts pd.DataFrames,". Or we remove this comment entirely, but both the order of indexset_1, indexset_2 as arguments for df_from_list() as well as the order of indexset.name, indexset.elements, ... in the function's return-value-DF are important for the test to work.

[meksor]

Ok, I'm kind of confused by that. Nevertheless i think this test might break randomly. (im surprised it works tbh). assert_unordered_equality exists because postgres will return items whichever way is the fastest. That means the same query might not always return a result in consistent order. That means there might actually not be a "correct order"...

[danielhuppmann]

Thanks @glatterf42 for rebasing and updating the PR. I think it's fine either way, whether the dimensionless unit comes pre-defined in a blank database or hast to be defined by a user. Whatever you prefer.

But I still think that the keyword argument should be unit instead of unit_or_name and it should take a ixmp4.Unit or a string. I believe that an argument unit_or_name creates more confusion than an argument unit that can handle multiple "forms" of a unit object.

[glatterf42]

Thanks, then I'll keep the dimensionless unit as pre-defined. The syntax changed now so that you can supply Scalars with unit, which can either be a str or a Unit object.

@meksor, please update your review as all your points should be addressed by now. The last thing from my side is adding another DB migration.

[glatterf42]

In fact, I don't think we need another migration here. There were some commits after d66a0a4 that affected data/db/optimization/scalar/model.py, but none that affected it in a way that changed the actual data model. When I ran alembic revision -m "Update optimization_scalar table" --autogenerate right now, it created a migration with an almost empty file, like

def upgrade():
    pass

def downgrade():
    pass

So I figure we don't need to push that migration here.

[glatterf42]

Rebased onto latest main.

[meksor]

Looks good thanks! Sorry for the delay!

[meksor]

Ok, I'm kind of confused by that. Nevertheless i think this test might break randomly. (im surprised it works tbh). assert_unordered_equality exists because postgres will return items whichever way is the fastest. That means the same query might not always return a result in consistent order. That means there might actually not be a "correct order"...