Add Backend, JDBCBackend, Model, GAMSModel classes

[khaeru]

As described in this comment on #78, this PR creates a Backend class separating the Python ixmp.core classes from the ixmp_source Java classes; and a Model class separating them from GAMS.

There is a corresponding PR iiasa/message_ix#249.

How to review

• Look at this diagram.
• Read the rest of this comment
• Run the tests.
• Look at the preview of the revised docs. Do these clearly describe how the pieces come together?
  • Core API: https://ixmp-khaeru.readthedocs.io/en/feature-backend/api-python.html
  • New Backend API: https://ixmp-khaeru.readthedocs.io/en/feature-backend/api-backend.html
  • New Model API: https://ixmp-khaeru.readthedocs.io/en/feature-backend/api-model.html
• Skim the code (‘Files changed’ tab).

Design

Why do this?

Several reasons…

1. It more clearly separates three things that were tangled up in the current core.py code:
   1. The promised behaviour (API) of the current Backend, i.e. ixmp_source.
   2. Any conversions between Python and Java not already handled by JPype.
   3. Actual functionality and logic of the ixmp.core classes themselves.
2. Via (iii), the functionality of ixmp becomes more obvious to users.
3. It can be clear where (i) is performing voodoo (b/c not open source) that is not apparent from reading (iii).
   • This could (doesn't need to be) a guide or motivation to moving code to the Python side, where it is (a) more transparent and (b) possibly more easily maintained.
4. Via (i), the barrier is much lower for users to write other backends, as discussed in Support databases other than JDBC #78 and in Slack.

Reading the code

• The current target here is to have zero changes in user-facing features; i.e., all tests pass, and no test has to be changed.
• There is a single Backend class that does not care very much about different classes of objects.
  • A hypothetical in-memory Backend might use a single object—an xarray.Dataset or even a simple dict()—and manipulate it in response to API calls.
  • The JDBCBackend is aware that ixmp_source contains many classes and manipulates them accordingly.
• (Outdated; this was an intermediate step.) I added some methods that spew noisy warnings, even though the tests pass. The warnings look like this:

  tests/test_core.py::test_init_scalar        
  tests/test_core.py::test_init_set                       
  tests/test_core.py::test_add_clone                                                                                                            
  tests/test_core.py::test_clone_edit                 
  tests/test_integration.py::test_run_clone           
  tests/test_integration.py::test_run_remove_solution
  tests/test_integration.py::test_multi_db_run                                                             
  tests/test_integration.py::test_multi_db_edit_source           
  tests/test_integration.py::test_multi_db_edit_target                                                                                         
    /home/khaeru/vc/iiasa/ixmp/ixmp/core.py:109: UserWarning: Accessing Platform._jobj in clone
  

  …and point to code that still needs to be refactored to work through the Backend. The PR will be ready once these warnings are all gone.
• (Outdated; I've also done this in the current PR at the encouragement of @gidden, below) The methods to_gdx() and read_sol_from_gdx() will, for now, force Backend to know what GAMS is. IMO this is less than ideal; per Support non-GAMS models #119, in the future these could be moved to a class GAMSModel(Model).

PR checklist

• Address remaining methods that access Java directly:
  • _jobj access in:
    • Platform.add_region, add_region_synomym, add_unit, check_access, clone, regions, scenario_list
    • TimeSeries.add_geodata, add_timeseries, check_out, commit, get_geodata, remove_geodata, remove_timeseries, timeseries
    • Scenario.add_timeseries, check_out, clone, commit, get_meta, remove_solution, set_meta
  • Scenario._item() calls in add_par, add_set, change_scalar, init_scalar, keys_for_quantity
  • ixmp.core._remove_ele
• Add 5 Backend methods required by message_ix.core.Scenario: ms_cat_set_elements, ms_cat_list, ms_cat_get_elements, ms_year_first_model, ms_years_active.
• Delete ixmp.Scenario.years_active → MsgScenario.getTecActYrs; this was incorrectly located in ixmp.Scenario rather than message_ix.Scenario.
• Add Model, GAMSModel classes, and transfer:
  • Scenario.to_gdx, .read_sol_from_gdx.
  • core.run_gams
• Any further changes required for message_ix to work — all commits after 896b3b8.
• Tests added.
• Documentation added.
• Release notes updated.

[zikolach]

@khaeru please see my 2 comments. I think we should have a meeting where you can present your vision, we can draw some diagrams to have common understanding of existing and planned architecture. I also commented on referenced issue #78 (comment)

[zikolach]

I didn't get the idea of having different methods for scenarios (s_-prefixed) and message scenarios (ms_-prefixed). It contradicts with original idea of ixmp package to be generic and potentially support many models.

[khaeru]

It contradicts with original idea of ixmp package to be generic and potentially support many models.

I agree. As I say in iiasa/message_ix#249, these are stack level violations where both ixmp and ixmp_source are currently not generic, but instead tailored to message_ix.

These were always there; this PR just makes it really obvious that they should not be there.

Our goal should be eventually to remove all ms_ methods.

• I already dealt with firstmodelyear and years_active by moving the logic into message_ix. The correspond ixmp_source methods are no longer used.
• The category/mapping set features should be moved back into ixmp.
• Backend (code like) ixmp_source should not treat ixmp.Scenario or its subclasses differently. Any special treatment of message_ix.Scenario should be in message_ix.Scenario.

[zikolach]

I see. Shouldn't then s_ prefixes be removed? As by default methods will operate on scenario (abstract API). Isn't it clear from parameter list and/or docstrings? Maybe also mark ms_ methods with deprecation comments as the code to be moved to message_ix package

[khaeru]

Isn't it clear from parameter list

It should be, I guess, yes 🤔 Okay, I will make that change before merging, then.

mark ms_ methods with deprecation comments

I think this code might end up in ixmp.Scenario, in which case it is just a name change rather than a deprecation and removal. But if this ends up going out as-is in the upcoming release, then I will add notices to explain that.

[zikolach]

I would not expose directly any references to JDBC essence. It should be hidden for consumer. E.g. in some future we may implement (java side) some additional storage options (nosql, file-based etc) which is not related to JDBC, but expose exactly same API (to python/R consumers).

[khaeru]

As mentioned in person, ixmp.Platform needs to choose and instantiate one Backend.

• For now, JDBCBackend is the only kind.
• Other Backends may provide storage without interacting with ixmp_source or any Java code.
• In the future, maybe, continued use of the "dbprops" and "dbtype" arguments could imply to use JDBCBackend, although I would prefer to clean up the semantics here.

[khaeru]

I think we should have a meeting where you can present your vision, we can draw some diagrams to have common understanding of existing and planned architecture.

Glad to do this. Who else besides us two should attend?

[khaeru]

Added a UML diagram to the PR description per discussions with @zikolach.

[khaeru]

Per a long discussion with @gidden and @zikolach today, I will merge this after @zikolach's approving review, and then open further issues for follow-up tasks including:

• Reconsider mapping sets code location and logic.

  The methods and backend calls can probably be moved from message_ix.Scenario to ixmp.Scenario.
  Logic that creates certain "internal sets" with names like "map_*" in message_ix can be implemented in Python, based on the R code in ixmp_source.

• Implement writing to/from GDX in Python.

  This is necessary to have other Backend implementations that do not depend on the Java code in ixmp_source and to break the strict dependency between GAMSModel and JDBCBackend. It does not necessarily mean we will remove the toGDX method in ixmp_source, since that is used by other code.

• Move more message_ix initialization code from Java ixmp_source to Python.

[zikolach]

Looks good for me!

I didn't do detailed python code review assuming most of the changes is just restructuring the code and API didn't break (as tests pass).

I would still consider keeping GAMS-related code on the Java side as to be used for:

• running integration tests (e.g. importing scenario (non-massage- specific on), running solver and writing it to DB)
• reusing code in web API backend application (e.g. export/import scenarios from scenario explorer)
• support for running GAMS-IXMP modeling from non-python environments (e.g. other JVM languages)

Thanks @khaeru for making changes, collecting feedback and keeping everyone informed!

[zikolach]

I see. Shouldn't then s_ prefixes be removed? As by default methods will operate on scenario (abstract API). Isn't it clear from parameter list and/or docstrings? Maybe also mark ms_ methods with deprecation comments as the code to be moved to message_ix package