WIP: Update parameter.add_data() functionality

[glatterf42]

As discovered in #108, we need a different implementation of parameter.add_data(). Ideally, we would like to make use of the JSONB type in-DB to avoid parsing the JSON field. However, with my current implementation, even that (which is only possible on postgres in the first place) is not at all faster than the pandas implementation (i.e. parsing the field to python, using pandas, writing it back).
I've tested this with a parameter with 1 indexset with 1000 values and with 60 indexsets with 1000 values (though the parameter was limited to 10000 values and units there): pandas is faster. Some numbers from my most recent test tun:

--------------------------------------------------------------------------------------- benchmark: 3 tests ---------------------------------------------------------------------------------------
Name (time in ms)                   Min                 Max                Mean            StdDev              Median               IQR            Outliers      OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_add_data[sqlite]           79.8390 (1.0)       79.8390 (1.0)       79.8390 (1.0)      0.0000 (1.0)       79.8390 (1.0)      0.0000 (1.0)           0;0  12.5252 (1.0)           1           1
test_add_data[postgres]        128.1381 (1.60)     128.1381 (1.60)     128.1381 (1.60)     0.0000 (1.0)      128.1381 (1.60)     0.0000 (1.0)           0;0   7.8041 (0.62)          1           1
test_add_data_json[sqlite]     276.1958 (3.46)     276.1958 (3.46)     276.1958 (3.46)     0.0000 (1.0)      276.1958 (3.46)     0.0000 (1.0)           0;0   3.6206 (0.29)          1           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

You'll notice that test_add_data_json[postgres] is missing: this test currently fails because it struggles to read in the column.names at one point. Hardcoding them does work, so it's doable, and you can test locally: it's still slower than pandas.

In general, _json indicated the variants working directly in-DB, while those without suffix are the pandas versions.

We might want to consider something like this if we want to use temporary tables in postgres.

Finally, for the 10000-values case, the in-DB workflow was not just slower: it didn't manage at all. My system did still work, but the test case wouldn't finish in more than 30 minutes. Pandas, on the other hand, took 2.8 seconds max (mean of 2.5).
So please, @danielhuppmann, let me know if using pandas is okay or if we should find a way to utilize JSONB (which would be nice for sure).
And if we go for JSONB, @meksor, please help me figure out the proper way to use it, i.e. one that's actually faster than pandas.

[danielhuppmann]

Thanks @glatterf42! I didn't have a chance to look at the code in detail, but if pulling all data and doing the comparison in memory is faster, that sounds like a good approach to me.

[glatterf42]

A while ago, Volker asked me to also benchmark upserting a parameter with roughly ten indexsets, but a million values. This gives a similar picture to before, though it may be a little worrying (I still have to look into it further): the only test that finished within an hour was the one using pandas and sqlite:

----------------------------------------------- benchmark: 1 tests -----------------------------------------------
Name (time in s)              Min      Max     Mean  StdDev   Median     IQR  Outliers     OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------------
test_add_data[sqlite]     96.3460  96.3460  96.3460  0.0000  96.3460  0.0000       0;0  0.0104       1           1
------------------------------------------------------------------------------------------------------------------

I'm not sure if 96 seconds is fine for us for this kind of operation, I'll benchmark the same operation with ixmp tomorrow.

When running on postgres, the benchmark produced this error:

self = <psycopg.Cursor [closed] [IDLE] (host=localhost user=postgres database=test) at 0x7f000bcd8290>
query = 'UPDATE optimization_parameter SET data=%(data)s::JSONB WHERE optimization_parameter.id = %(optimization_parameter_id)s::INTEGER'
params = {'data': Jsonb({'Indexset 0': [0, 0, 0, 0, 0, 0, 0 ... (128780213 chars)), 'optimization_parameter_id': 1}

    def execute(
        self,
        query: Query,
        params: Optional[Params] = None,
        *,
        prepare: Optional[bool] = None,
        binary: Optional[bool] = None,
    ) -> Self:
        """
        Execute a query or command to the database.
        """
        try:
            with self._conn.lock:
                self._conn.wait(
                    self._execute_gen(query, params, prepare=prepare, binary=binary)
                )
        except e._NO_TRACEBACK as ex:
>           raise ex.with_traceback(None)
E           sqlalchemy.exc.OperationalError: (psycopg.errors.ProgramLimitExceeded) total size of jsonb object elements exceeds the maximum of 268435455 bytes
E           CONTEXT:  unnamed portal parameter $1 = '...'
E           [SQL: UPDATE optimization_parameter SET data=%(data)s::JSONB WHERE optimization_parameter.id = %(optimization_parameter_id)s::INTEGER]
E           [parameters: {'data': Jsonb({'Indexset 0': [0, 0, 0, 0, 0, 0, 0 ... (128780213 chars)), 'optimization_parameter_id': 1}]
E           (Background on this error at: https://sqlalche.me/e/20/e3q8)

.venv/lib/python3.12/site-packages/psycopg/cursor.py:732: OperationalError

Meanwhile, when using the in-DB _json functionality, the sqlite test wouldn't finish within an hour (so that I couldn't even start the one on postgres, though I doubt it would fare much better).

[glatterf42]

Running the same benchmark (upserting a million values to one parameter with 12 indexsets) in ixmp takes just 53.48 seconds, so the current pandas-implementation in ixmp4 would actually constitute a downgrade, it seems.
Hence, I should invest more time into making the new implementation more efficient. Since I tried that before and wasn't too successful, I'd appreciate any help or guidance you can give, @meksor.