feat(exasol): add exasol backend #7303

Nicoretti · 2023-10-06T12:16:25Z

👋 Hi Ibis Team,

I reached out to you a few months ago (here),
but I wasn't able to follow up on this in a more timely manner.

This PR contains an initial draft of adding an Exasol backend to Ibis.
I am aware that the PR is not very polished yet and also lacks a significant number of features.
That said, with the PR, I also wanted to establish a means of communication to get some feedback
and align with you while pushing the feature of an Exasol backend forward.

Status Quo

An Exasol backend for testing is set up
- Starting the database
- Initializing data
- ...
A bare minimum SQLA-based backend is implemented
A bare minimum of documentation is added
Tests can be run
- just up exasol
- just test exasol
  - 715 failed, 430 passed, 38 skipped, 26312 deselected, 38 xfailed, 33 errors

Questions

Questions for/to the Ibis Project

What features does the Ibis Project require for the MVP for a new backend?
How do I update/work with the Nix shell when dependencies, etc. are added?
- nix-shell failed after introducing new dependencies.
- Is there a README or some other source that can help here (except for learning Nix 😅)?

To-Do's

Edited:

   --- Removed ---

It would be great if you could give me some initial feedback on the PR and have a look at
my questions. I appreciate your effort and look forward to hearing from you.

best
Nico

cpcloud

Thanks for the PR! Glad to hear back from you!

Gave this a once over. Overall, this looks pretty good :)

docker/exasol/Dockerfile

docker-compose.yml

pyproject.toml

ibis/backends/exasol/tests/conftest.py

cpcloud · 2023-10-06T12:39:17Z

To answer some of your questions:

What features does the Ibis Project require for the MVP for a new backend?

We don't really have a documented bare minimum feature set for new backends.

If I had to say what it was, I think it'd be:

con.table(name) returns a table expression without failing
con.table(name).execute() returns a pandas DataFrame without failing

Basically, you should be able to run the equivalent of SELECT * FROM t and bring the results back into Python as a pandas DataFrame.

How do I update/work with the Nix shell when dependencies, etc. are added?

We don't have specific documentation for this. I can add some.

I suspect you need to run just lock, and then direnv reload (if you're using direnv) or exit and re-enter the nix-shell.

cpcloud · 2023-10-06T12:44:08Z

Regarding getting the tests to pass, the least painful strategy is to insert

pytestmark = pytest.mark.notimpl(["exasol"])

at the top of every test_*.py file (or insert "exasol" into the existing pytestmark's list) and then start implementing functionality.

When a test that is marked as notimpl passes, you'll see a pytest failure with a message like this:

[XPASS(strict)] ...

Indicating that you've successfully passed the test. You can then remove "exasol" from the notimpl list.

Nicoretti · 2023-10-08T13:31:49Z

Hi @cpcloud,

thanks for all the input and help, I'll pickup on it tomorrow.

best
Nico

Nicoretti · 2023-10-09T11:34:10Z

@cpcloud I assume the final PR will be squash merged and therefore did not follow the projects commit message conventions.
If that assumption is wrong, please let me know.

cpcloud · 2023-10-09T11:57:57Z

@Nicoretti We can clean up the commits once everything is green :) No need to worry about it for now.

docker-compose.yml

ibis/backends/base/__init__.py

Nicoretti · 2023-10-11T08:51:45Z

Hi @cpcloud

Just did another catchup with your master. I am wondering if the following workflow for addressing merge conflicts in: poetry.lock, requirements-dev.txt, poetry-overrides.nix is correct:

Make sure pyproject.toml is correct (correctly merged)
Remove the conflicted poetry.lock
Run poetry lock or poetry update
Run just lock

cpcloud · 2023-10-11T09:03:20Z

@Nicoretti That looks right!

The only thing I would change is that you should be able to merge steps 3 and 4 into

just lock

Nicoretti · 2023-10-11T16:20:57Z

@cpcloud sry had to force push. I had rebased the changes from the ibis/master into this branch and pushed it 😅.

btw doing a "clean" lock, does not seem to work properly (even on ibis/master). Poetry does not finish resolving (conflicts).
A relock, with an existing poetry.lock is working fine though.

"Clean" Lock

rm poetry.lock
just lock

cpcloud · 2023-10-17T09:59:05Z

@Nicoretti I am going to rebase this PR and fix up the merge conflicts!

Nicoretti · 2023-10-17T10:26:10Z

@cpcloud thx for updating

cpcloud · 2023-10-20T09:54:36Z

I can help with the poetry.lock conflicts!

Nicoretti · 2023-10-20T10:13:00Z

@cpcloud

I can help with the poetry.lock conflicts.
Thanks appreciate it, next time I'll give it another try ;).

To spare you some time
To enable myself (I think, I understood what went wrong the last time)

Side Note(s):

Sorry for the slow progress, have been dealing with other things not in my direct control
Next week I won't be able to work on the PR but I'll pick up on it the week after

best
Nico

ibis/backends/exasol/__init__.py

Nicoretti · 2023-11-07T10:36:00Z

Hi @cpcloud,

I am back on the PR. Sorry for the extra delay; I had to address a bug and some other issues in other projects I am maintaining.

Since my catch-up yesterday, I have experienced two issues. I'm just wondering if you may have any idea regarding them.

The nix-shell does not seem to work the way described here anymore. This is probably due to PR #7437. Is this intended?

Executing tests seems pretty flaky (not only in parallel mode):

pytest -n 0 --dist no -m exasol

749 failed, 233 passed, 34 skipped, 27246 deselected, 240 xfailed, 24 errors in 350.39s (0:05:50)
561 failed, 434 passed, 34 skipped, 27246 deselected, 242 xfailed, 9 errors in 442.03s (0:07:22)
618 failed, 377 passed, 34 skipped, 27246 deselected, 242 xfailed, 9 errors in 465.57s (0:07:45)

pytest -n 0 --dist loadgroup -m exasol

582 failed, 413 passed, 34 skipped, 27246 deselected, 242 xfailed, 9 errors in 469.88s (0:07:49)
320 failed, 227 passed, 34 skipped, 27246 deselected, 153 xfailed, 546 errors in 323.05s (0:05:23)
626 failed, 367 passed, 34 skipped, 27246 deselected, 242 xfailed, 11 errors in 435.29s (0:07:15)

pytest -n auto -m exasol

512 failed, 483 passed, 34 skipped, 242 xfailed, 9 errors in 323.58s (0:05:23)
518 failed, 478 passed, 34 skipped, 241 xfailed, 9 errors in 279.37s (0:04:39)
443 failed, 415 passed, 34 skipped, 200 xfailed, 188 errors in 286.54s (0:04:46)

just test exasol

533 failed, 462 passed, 34 skipped, 242 xfailed, 9 errors in 253.28s (0:04:13)
518 failed, 478 passed, 34 skipped, 241 xfailed, 9 errors in 273.90s (0:04:33)
545 failed, 448 passed, 34 skipped, 243 xfailed, 10 errors in 268.66s (0:04:28)

I was wondering if this rings a bell or sounds like a problem you've already encountered in the past with other backends.

Thanks for being so supportive all along.

Best,
Nico

gforsyth · 2023-11-07T18:08:45Z

Hey @Nicoretti -- @cpcloud is on a well-deserved vacation but will be back next week.

You are right that those nix instructions are out-of-date. You have two options if you want to use nix (and I'll update the docs):

Run nix develop which will drop you into a shell that has all of the packages loaded up
Use direnv (here: https://direnv.net/docs/installation.html) and it will auto-load things into your current shell environment when you enter the ibis directory (@cpcloud and I both use direnv)

Regarding test flakiness, that could be a bunch of things -- some of the failure modes might help diagnose. If I had to guess, some tests are leaving some global state around, and since we use pytest-randomly the tests will always run in a different order, so you might get "lucky" and have tests pass accidentally some of the time.

One thing to try initially is to add --randomly-dont-reorganize to your pytest call and see if the number of failures stays constant (at least in a serial run).

Nicoretti · 2023-11-08T07:40:16Z

Hi @gforsyth,

Thanks for the timely update and feedback, appreciate it. The random test execution is a very good point, I didn't really had it on my radar yet.

Best,
Nico

ibis/backends/base/sql/alchemy/__init__.py

ibis/backends/exasol/tests/conftest.py

cpcloud · 2023-11-21T10:02:14Z

I am fixing the merge conflicts!

cpcloud · 2023-11-21T10:06:52Z

@Nicoretti Is there any chance sqlalchemy-exasol can be made compatible with the latest sqlalchemy release that is <2 (1.4.50)?

Python 3.12 support is coming up and the sqlalchemy version that's supported by sqlalchemy-exasol is now constraining us to a version that doesn't provide binary wheels for python 3.12.

cpcloud · 2023-11-21T10:09:56Z

Generally speaking it should be fine to set sqlalchemy constraints to allow patch version bumps, they tend not to break things in patch releases, so instead of the <1.4.45 bound that's in sqlalchemy-exasol it would be fine to make that <1.5 if you're not comfortable making it <2.

cpcloud · 2023-11-21T10:16:58Z

I am running the test suite now and will help grind through the current failures. I'd like to avoid long running PRs if possible 😅

Nicoretti · 2023-11-21T10:20:18Z

Generally speaking it should be fine to set sqlalchemy constraints to allow patch version bumps, they tend not to break things in patch releases, so instead of the <1.4.45 bound that's in sqlalchemy-exasol it would be fine to make that <1.5 if you're not comfortable making it <2.

@cpcloud Generally, yes. I need to re-validate it against the test suite of that sqlalchemy version though. I'll take care of it this week. I hope that fits your schedule.

cpcloud · 2023-11-21T10:23:21Z

@Nicoretti It looks like your approach so far is to go through every test file and implement functionality until tests pass.

I highly recommend a different approach that will get this merged upstream much faster:

Mark everything that is currently failing as pytest.mark.notimpl and create follow-up pull requests to add functionality.

This also has the side benefit of user-facing release notes that show when particular functionality was added.

cpcloud · 2023-11-21T10:23:57Z

Generally speaking it should be fine to set sqlalchemy constraints to allow patch version bumps, they tend not to break things in patch releases, so instead of the <1.4.45 bound that's in sqlalchemy-exasol it would be fine to make that <1.5 if you're not comfortable making it <2.

@cpcloud Generally, yes. I need to re-validate it against the test suite of that sqlalchemy version though. I'll take care of it this week. I hope that fits your schedule.

Sounds good to me!

gforsyth · 2023-12-08T20:32:43Z

ibis/backends/base/__init__.py

@@ -52,6 +52,8 @@
    # druid allows double quotes for identifiers, like postgres:
    # https://druid.apache.org/docs/latest/querying/sql#identifiers-and-literals
    "druid": "postgres",
+    # closest match see https://github.com/ibis-project/ibis/pull/7303#discussion_r1350223901
+    "exa.websocket": "oracle",


Here you state that oracle is the closest matching sqlglot dialect, but all the sqlglot calls in exasol/__init__.py are using the postgres dialect. I think one of these needs to change -- should this be postgres here? Or should the dialect= kwargs be oracle?

Unfortunately, I can't provide a link to my previous conversation with @cpcloud due to GitHub's limitation in displaying outdated comments during force rebasing.

To summarize our discussion: historically, @cpcloud generously assisted me by providing implementations using postgress for certain code segments. Later, we delved into identifying the closest SQL dialect supported by sqlglot. During this, I consulted our DB developers to determine the most similar dialect.

In short: Oracle appears to be the closest match, as per their insights. However, navigating dialects is complex, especially with Exasol, where support for specific syntax constructs varies across dialects. Hence, while there might not be an official closest match, the consensus from our DB developers leans toward Oracle.

ibis/backends/tests/test_generic.py

gforsyth

This looks pretty much ready to go, @Nicoretti ! Thanks for all of the work on this!

There's the one question about which sqlglot dialect to use that we should resolve before this gets merged.

I'd prefer if we didn't add so many strict=False tests, but sometimes that is just what needs to happen. Are there any hints as to the cause of the flakiness?

Nicoretti · 2023-12-11T07:37:35Z

@gforsyth

Are there any hints as to the cause of the flakiness?

One issues which caused failing tests in serial and parallel execution is fixed and noted in
the create_schema function (see snippet below).

...
    # Exasol implicitly opens the created schema, therefore we need to restore
    # the previous context.
    action = (
        sa.text(f"OPEN SCHEMA {open_schema}")
        if open_schema
        else sa.text(f"CLOSE SCHEMA {name}")
    )
    con.exec_driver_sql(action)

As of today, when executed serially, the tests should not be flaky. However, concerning parallel execution, there still seem to be one or more issues that need identification and addressing.

Nicoretti · 2023-12-11T08:49:58Z

Having rebased against, the latest master a backend test (flink) is failing during install.
I have seen similar things in the past, it potentionaly have been an network issues during the pip install.

Nicoretti · 2023-12-12T09:49:56Z

Just rebased, oracle keeps failing, not entirely sure why this is (how it relates to my changes).
Any ideas?

cpcloud · 2023-12-12T11:19:08Z

Bringing in the upstream changes from

https://github.com/ibis-project/ibis/compare/7f41bf29e38665c9a2cd323e4a6e09d144e22e2e..90aee1a05296526ee1a034327699e5bea0a3cb8b#diff-0090401604eb2e0aa1e71305358a1c922ea42f8119a64c91f9e5038b58e73984L233

broke the oracle tests.

I had to change the type inference related code for oracle in PR due to the change to gen_name. In the meantime, there was a change to this bit of code upstream to fix a heisenbug.

I'll fix it back up.

cpcloud · 2023-12-12T11:29:34Z

It looks like you also reverted https://github.com/ibis-project/ibis/compare/90aee1a05296526ee1a034327699e5bea0a3cb8b..fc7c9a6e46a2bc769df583d0f044c22146daee9b#diff-70e8f790a7cd8392a8b812c47a95be84f3835051e37a873536cefb3cb726d360L97

which is an important change to ensure that we don't break BigQuery due to Exasol's inability to create a table whose name starts with _.

I'll add this back too.

Nicoretti · 2023-12-12T12:34:19Z

Bringing in the upstream changes from

https://github.com/ibis-project/ibis/compare/7f41bf29e38665c9a2cd323e4a6e09d144e22e2e..90aee1a05296526ee1a034327699e5bea0a3cb8b#diff-0090401604eb2e0aa1e71305358a1c922ea42f8119a64c91f9e5038b58e73984L233

broke the oracle tests.

I had to change the type inference related code for oracle in PR due to the change to gen_name. In the meantime, there was a change to this bit of code upstream to fix a heisenbug.

I'll fix it back up.

Thanks for your help! Heisen bugs ... had my fair share of those 😅.

It looks like you also reverted https://github.com/ibis-project/ibis/compare/90aee1a05296526ee1a034327699e5bea0a3cb8b..fc7c9a6e46a2bc769df583d0f044c22146daee9b#diff-70e8f790a7cd8392a8b812c47a95be84f3835051e37a873536cefb3cb726d360L97

which is an important change to ensure that we don't break BigQuery due to Exasol's inability to create a table whose name starts with _.

I'll add this back too.

Thanks, my bad. I assumed that I have accidentally overwritten changes from the upstream (master) and therefore reverted some changes of mine to the latest master.

gforsyth · 2023-12-12T15:36:16Z

docs/backends/exasol.qmd

+Install for Exasol:
+
+```{.bash}
+conda install -c conda-forge ibis-exasol


@Nicoretti -- I will try to remember to add this to the conda-forge recipe when we do our next release, but you should feel free to ping me to remind me.

Also, if you can add sqlalchemy-exasol to conda-forge that will greatly improve the turnaround time on getting ibis-exasol added.

@gforsyth

@Nicoretti -- I will try to remember to add this to the conda-forge recipe when we do our next release, but you should feel free to ping me to remind me.

Removing it from the docs would be a possibility too. I just used another backend documentation as template, wasn't aware about all implications. But if we want to keep it, I can over to create a ticket as reminder.

Also, if you can add sqlalchemy-exasol to conda-forge that will greatly improve the turnaround time on getting ibis-exasol added.

Previously, sqlalchemy-exasol was available as a conda-forge package, but it was discontinued, possibly due to being outdated and underused. I need to determine if we should reintroduce it. Given that we now have a concrete use case with Ibis, it seems more likely that its reintroduction would be accepted.

I can help maintain the feedstock (what's one more on a big pile 😁?) -- I think it's a good idea to have it available on conda-forge, if only so that exasol isn't the only backend that you can't install with conda/mamba.

recipe PR here: conda-forge/staged-recipes#24777

nevermind! here: https://github.com/conda-forge/sqlalchemy_exasol-feedstock

the underscore threw me when I was looking before. looks like the conda-forge package is available and up-to-date, so that part is taken care of.

gforsyth · 2023-12-12T15:38:48Z

What's going on with the coverage warnings in exasol/__init__.py? Are we failing to upload something? Because there's no way the test suite is running and we aren't hitting _get_sqla_table

Nicoretti · 2023-12-13T08:17:49Z

What's going on with the coverage warnings in exasol/__init__.py? Are we failing to upload something? Because there's no way the test suite is running and we aren't hitting _get_sqla_table

@gforsyth
Good question, I just validated _get_sqla_table (ibis/backends/exasol/__init__.py#L147-L148) gets hit repeatedly (when the suit is executed locally). Not sure on the sonar cloud part, I don't have to much experience with it yet.

gforsyth · 2023-12-13T15:25:07Z

@gforsyth
Good question, I just validated _get_sqla_table (ibis/backends/exasol/__init__.py#L147-L148) gets hit repeatedly (when the suit is executed locally). Not sure on the sonar cloud part, I don't have to much experience with it yet.

Looks like @cpcloud fixed it!

cpcloud · 2023-12-13T19:43:39Z

@gforsyth This LGTM! Anything else left to do?

gforsyth

This is great @Nicoretti! Nothing blocking, just a few notes and we can tackle them in a follow-up, too.

ibis/backends/base/__init__.py

gforsyth · 2023-12-13T17:35:54Z

ibis/backends/tests/test_client.py

@@ -251,6 +251,7 @@ def tmpcon(alchemy_con):
    ["mssql"],
    reason="mssql supports support temporary tables through naming conventions",
 )
+@mark.notimpl(["exasol"], reason="Exasol does not support temporary tables")


Suggested change

@mark.notimpl(["exasol"], reason="Exasol does not support temporary tables")

@mark.notyet(["exasol"], reason="Exasol does not support temporary tables")

I think this should be a notyet or a never -- notimpl is a signal to us that Ibis hasn't added support for something that is currently possible.

This currently would be a never because it is not supported by Exasol and I don't know if there are any plans to do so in the near future.

gforsyth · 2023-12-13T17:37:54Z

ibis/backends/tests/test_generic.py

+        None,
+        param(
+            [],
+            marks=pytest.mark.notimpl(["exasol"], raises=ExaQueryError, strict=False),


What's happening here? Is this test flaky?

From what I can tell exasol-substet1-all is reliably failing, due to the lack of support of ? within the where clause:

pyexasol.exceptions.ExaQueryError: ( message => where clause must be a boolean expression (Session: 1785240712777826304) dsn => localhost:8563 user => sys schema => EXASOL session_id => 1785240712777826304 code => 42000 query =>

WITH t0 AS (SELECT t2."id" AS "id", t2."bool_col" AS "bool_col", t2."tinyint_col" AS "tinyint_col", t2."small_int" AS "small_int", t2."int_col" AS "int_col", t2."bigint_col" AS "bigint_col", t2."float_col" AS "float_col", t2."double_col" AS "double_col", t2."date_string_col" AS "date_string_col", t2."string_col" AS "string_col", t2."timestamp_col" AS "timestamp_col", t2."year" AS "year", t2."month" AS "month", CASE WHEN (t2."int_col" = ?) THEN NULL ELSE t2."float_col" END AS col_1, CASE WHEN (t2."int_col" = ?) THEN NULL ELSE t2."float_col" END AS col_2, CASE WHEN (t2."int_col" = ? OR t2."int_col" = ?) THEN NULL ELSE t2."float_col" END AS col_3 FROM functional_alltypes AS t2) SELECT t1.col_1, t1.col_2, t1.col_3 FROM (SELECT t0.col_1 AS col_1, t0.col_2 AS col_2, t0.col_3 AS col_3 FROM t0) AS t1 WHERE ? )

That said, I don't know if this only is a limitation of the web socket based driver/protocol, or rather a general one.
@gforsyth I can look more into the details If you want me too.

Side Note:

SELECT Foo, Bar from T WHERE Foo = ?;

Should work though.

gforsyth · 2023-12-13T17:41:22Z

ibis/backends/tests/test_temporal.py

@@ -2447,6 +2505,7 @@ def test_date_scalar_from_iso(con):
    raises=sa.exc.DatabaseError,
    reason="ORA-22849 type CLOB is not supported",
 )
+@pytest.mark.notimpl(["exasol"], raises=AssertionError, strict=False)


is this also passing sometimes?

I think you are maybe on to something here. Just removed the marker locally and ran the test multiple times:

10x test_temporal suit only

Always Green

5x Full backend test suite

Always green

…t an underscore

…` executable

Co-authored-by: Gil Forsyth <gforsyth@users.noreply.github.com>

cpcloud · 2023-12-14T09:51:32Z

Bombs away! Thanks for your work @Nicoretti, look forward to working with you more on this!

Nicoretti · 2023-12-14T10:15:17Z

Bombs away! Thanks for your work @Nicoretti, look forward to working with you more on this!

Looking forward to keep working with you too!

cpcloud reviewed Oct 6, 2023

View reviewed changes

cpcloud added the new backend PRs or issues related to adding new backends label Oct 6, 2023

Nicoretti commented Oct 9, 2023

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

ibis/backends/base/__init__.py Outdated Show resolved Hide resolved

cpcloud added the feature Features or general enhancements label Oct 12, 2023

Nicoretti commented Oct 20, 2023

View reviewed changes

ibis/backends/exasol/__init__.py Outdated Show resolved Hide resolved

Nicoretti commented Nov 8, 2023

View reviewed changes

ibis/backends/base/sql/alchemy/__init__.py Outdated Show resolved Hide resolved

Nicoretti commented Nov 14, 2023

View reviewed changes

ibis/backends/exasol/tests/conftest.py Outdated Show resolved Hide resolved

cpcloud added the experimental Experimental features label Nov 21, 2023

gforsyth reviewed Dec 8, 2023

View reviewed changes

ibis/backends/tests/test_generic.py Outdated Show resolved Hide resolved

gforsyth reviewed Dec 8, 2023

View reviewed changes

Nicoretti requested a review from gforsyth December 11, 2023 07:44

gforsyth reviewed Dec 12, 2023

View reviewed changes

cpcloud approved these changes Dec 13, 2023

View reviewed changes

gforsyth approved these changes Dec 13, 2023

View reviewed changes

Nicoretti and others added 7 commits December 14, 2023 04:21

feat(exasol): add exasol backend

041d93d

chore(oracle): port fix to sqlglot impl

6045bf6

chore(bigquery): ensure that bigquery memtable names are valid withou…

6e94e2d

…t an underscore

test(exasol): only search in /usr in the container for the `exaplus…

50b1a6a

…` executable

chore(deps): relock

1a22164

chore(exasol): rename sqlglot dialect mapping

5de0749

Co-authored-by: Gil Forsyth <gforsyth@users.noreply.github.com>

test(dot-sql): enable transpilation of dialects to exasol

2f0bbdc

cpcloud merged commit a623456 into ibis-project:master Dec 14, 2023
91 checks passed

Nicoretti deleted the exasol-backend branch December 14, 2023 10:15

This was referenced Jan 19, 2024

feat: Exasol backend #8037

Closed

meta: New backends #8035

Closed

	@mark.notimpl(["exasol"], reason="Exasol does not support temporary tables")
	@mark.notyet(["exasol"], reason="Exasol does not support temporary tables")

feat(exasol): add exasol backend #7303

feat(exasol): add exasol backend #7303

Conversation

Nicoretti commented Oct 6, 2023 • edited Loading

Status Quo

Questions

Questions for/to the Ibis Project

Other Questions

To-Do's

cpcloud left a comment

Choose a reason for hiding this comment

cpcloud commented Oct 6, 2023

cpcloud commented Oct 6, 2023 • edited Loading

Nicoretti commented Oct 8, 2023

Nicoretti commented Oct 9, 2023 • edited Loading

cpcloud commented Oct 9, 2023

Nicoretti commented Oct 11, 2023 • edited Loading

cpcloud commented Oct 11, 2023

Nicoretti commented Oct 11, 2023 • edited Loading

"Clean" Lock

cpcloud commented Oct 17, 2023

Nicoretti commented Oct 17, 2023 • edited Loading

cpcloud commented Oct 20, 2023

Nicoretti commented Oct 20, 2023

Nicoretti commented Nov 7, 2023

gforsyth commented Nov 7, 2023

Nicoretti commented Nov 8, 2023 • edited Loading

cpcloud commented Nov 21, 2023

cpcloud commented Nov 21, 2023

cpcloud commented Nov 21, 2023 • edited Loading

cpcloud commented Nov 21, 2023

Nicoretti commented Nov 21, 2023

cpcloud commented Nov 21, 2023

cpcloud commented Nov 21, 2023

Choose a reason for hiding this comment

Nicoretti Dec 11, 2023 • edited Loading

Choose a reason for hiding this comment

gforsyth left a comment

Choose a reason for hiding this comment

Nicoretti commented Dec 11, 2023 • edited Loading

Nicoretti commented Dec 11, 2023

Nicoretti commented Dec 12, 2023

cpcloud commented Dec 12, 2023

cpcloud commented Dec 12, 2023

Nicoretti commented Dec 12, 2023

Choose a reason for hiding this comment

Nicoretti Dec 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gforsyth commented Dec 12, 2023

Nicoretti commented Dec 13, 2023 • edited Loading

gforsyth commented Dec 13, 2023

cpcloud commented Dec 13, 2023

gforsyth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nicoretti Dec 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Dec 14, 2023

Nicoretti commented Dec 14, 2023

Nicoretti commented Oct 6, 2023 •

edited

Loading

cpcloud commented Oct 6, 2023 •

edited

Loading

Nicoretti commented Oct 9, 2023 •

edited

Loading

Nicoretti commented Oct 11, 2023 •

edited

Loading

Nicoretti commented Oct 11, 2023 •

edited

Loading

Nicoretti commented Oct 17, 2023 •

edited

Loading

Nicoretti commented Nov 8, 2023 •

edited

Loading

cpcloud commented Nov 21, 2023 •

edited

Loading

Nicoretti Dec 11, 2023 •

edited

Loading

Nicoretti commented Dec 11, 2023 •

edited

Loading

Nicoretti Dec 13, 2023 •

edited

Loading

Nicoretti commented Dec 13, 2023 •

edited

Loading

Nicoretti Dec 14, 2023 •

edited

Loading