[Python/SQLAlchemy] Add example programs about efficient bulk inserts with SQLAlchemy, pandas, and Dask #64

amotl · 2023-05-11T10:21:34Z

Hi there,

this patch carries over the example programs about Python/SQLAlchemy/pandas/Dask from crate/crate-python#552. Originally, the corresponding programs have been conceived as individual gists:

With kind regards,
Andreas.

/cc @marijaselakovic, @karynzv, @WalBeh, @hammerhead, @hlcianfagna, @andnig, @ckurze

Backlog

~~Software tests and Dependabot configuration works well for the other code example directories, and will be added here as well.~~ Done, with a code coverage of 95%.
~~Add fast-path INSERT method insert_bulk for SQLAlchemy/pandas/Dask crate-python#553 needs to be merged and released beforehand, to make the examples actually work.~~ Merged in July 2023.
Some adjustments may be added to bring all programs into the same shape.
Other than demonstrating write operations, also demonstrate read operations?

=> Backlog items have been added to GH-95, in order to finally get this merged, so it can be promoted and re-used.

amotl · 2023-10-28T15:26:22Z

.github/workflows/test-python-sqlalchemy.yml

+      - name: Validate by-language/python-sqlalchemy
+        run: |
+          python testing/ngr.py --accept-no-venv by-language/python-sqlalchemy


We finally added a universal test runner, currently located at testing/ngr.py, which is effectively just wrapping a few other calls, to be able to start maintaining a concise incantation syntax across different CI recipes.

Because it is written in Python, and aims to be generic, without assuming it is invoked on any kind of CI system, it has another benefit that we can mirror incantation style and experience between CI systems vs. developer sandbox operations, which significantly eases administration, because the developer can easily run the same CI recipe locally without much efforts, DWIM-style.

amotl · 2023-10-28T15:27:14Z

testing/ngr.py

+"""
+-----
+About
+-----
+
+Next Generation Runner (ngr): Effortless invoke programs and test harnesses.
+
+-------
+Backlog
+-------
+- After a few iterations, refactor to separate package.


ngr.py will only be incubated here, it will be refactored to a standalone package after a few iterations. If you like the idea, it can be re-used on other projects, for example on crate-qa, after it learned to invoke relevant test suites of a few other programming languages.

testing/ngr.py moved into pueblo.ngr already, and learned three more directory shapes to invoke: javascript, make, and rust.

class ItemType(Enum): DOTNET = "dotnet" JAVA = "java" JAVASCRIPT = "javascript" MAKE = "make" PHP = "php" PYTHON = "python" RUBY = "ruby" RUST = "rust"

pyveci/pueblo@b7a25308ace20

amotl · 2023-10-28T16:04:14Z

by-language/python-sqlalchemy/README.rst

+*****
+About
+*****
+
+Example programs demonstrating CrateDB's SQLAlchemy adapter and dialect.


Improve documentation I

Currently, the example programs added by this patch are mostly about pandas/Dask operations through SQLAlchemy. This is a special topic, where we elaborated extensively about, at ¹. Here, we surely want to link to this resource.

Footnotes

https://cratedb.com/docs/python/en/latest/by-example/sqlalchemy/dataframe.html ↩

The README has been improved, see its rendered version here.

-- https://github.com/crate/cratedb-examples/blob/amo/python-sqlalchemy/by-language/python-sqlalchemy/README.rst

amotl · 2023-10-28T16:21:02Z

by-language/python-sqlalchemy/README.rst

+*****
+Usage
+*****
+


Improve documentation II

In this case, within the Usage section, it should clearly be focused on "library use" details, like, how to exactly use the insert_bulk function with pandas and Dask.

from crate.client.sqlalchemy.support import insert_bulk # pandas df.to_sql(name=self.table_name, con=engine, if_exists="append", index=False, chunksize=bulk_size, method=insert_bulk) # Dask ddf.to_sql("testdrive", uri=DBURI, index=False, if_exists="replace", chunksize=10_000, parallel=True, method=insert_bulk)

What is currently within the Usage section, should go into an Examples section instead.

Fixed/improved, see above. An example has been added.

amotl · 2023-10-28T16:29:32Z

by-language/python-sqlalchemy/README.rst

+    # Connect to CrateDB Cloud.
+    time python insert_pandas.py --dburi='crate://admin:<PASSWORD>@example.aks1.westeurope.azure.cratedb.net:4200?ssl=true'


This is a bit poor. Improve/highlight operations with CrateDB Cloud.

Dito. Fixed.

amotl · 2023-10-28T16:30:09Z

by-language/python-sqlalchemy/README.rst

+    source .venv/bin/activate
+    pip install --upgrade --requirement requirements.txt


A test suite has been added. Document how it can be invoked after installing the requirements.

pytest

amotl · 2023-10-28T16:31:21Z

by-language/python-sqlalchemy/insert_pandas.py

+
+logger = logging.getLogger(__name__)
+
+pkg_resources.require("sqlalchemy>=2.0")


Also reflect this within the requirements.txt file, and probably also within insert_dask.py.

Is it still needed at all then?

seut · 2023-10-30T14:24:04Z

by-language/python-sqlalchemy/insert_efficient.py

+    # Run PostgreSQL
+    docker run --rm -it --publish=5432:5432 --env "POSTGRES_HOST_AUTH_METHOD=trust" postgres:15 postgres -c log_statement=all
+
+    # Use SQLite
+    time python insert_efficient.py sqlite multirow
+    time python insert_efficient.py sqlite batched
+
+    # Use PostgreSQL
+    time python insert_efficient.py postgresql multirow
+    time python insert_efficient.py postgresql batched


How is this example related to other DB's? Why should I run this for other DB's?

I wanted to have an easy stage to run comparisons side-by-side. It is now integrated into the relevant example program, and I find it convenient.

When using it with higher numbers of records, you can also get a good feeling about insert performance compared with the other DBs.

seut · 2023-10-30T14:24:59Z

by-language/python-sqlalchemy/README.rst

+
+To start a CrateDB instance on your machine for evaluation purposes, invoke::
+
+    docker run -it --rm --publish=4200:4200 --publish=5432:5432 crate


Is publishing the PG port 5432 required for this example?

Erm. Probably not. It is just my canonical command I slap everywhere, not needing to discriminate between different variants, and also advertising a bit.

seut · 2023-10-30T14:30:19Z

by-language/python-sqlalchemy/insert_pandas.py

+
+@click.command()
+@click.option("--dburi", type=str, default="crate://localhost:4200", required=False, help="SQLAlchemy database connection URI.")
+@click.option("--mode", type=str, default="bulk", required=False, help="Insert mode. Choose one of basic, multi, bulk.")


could use a choice option to make possible options strict.

Suggested change

@click.option("--mode", type=str, default="bulk", required=False, help="Insert mode. Choose one of basic, multi, bulk.")

@click.option("--mode", type=click.Choice(['basic', 'multi', 'bulk']), default="bulk", required=False, case_sensitive=False, help="Insert mode. Choose one of basic, multi, bulk.")

Did you ask another expert on this?

TypeError: Parameter.__init__() got an unexpected keyword argument 'case_sensitive'

amotl mentioned this pull request May 11, 2023

Documentation: Add executable code examples crate/crate-python#552

Closed

amotl force-pushed the amo/python-sqlalchemy branch from 06366e3 to dcf601d Compare May 11, 2023 10:58

amotl changed the title ~~[Python/SQLAlchemy] Add two example programs~~ [Python/SQLAlchemy] Add example programs about efficient bulk inserts with SQLAlchemy, pandas, and Dask May 11, 2023

amotl force-pushed the amo/python-sqlalchemy branch from dcf601d to 8dbf13b Compare May 11, 2023 11:04

amotl mentioned this pull request May 11, 2023

Documentation: Add pointers to executable code examples crate/crate-python#554

Merged

amotl force-pushed the amo/python-sqlalchemy branch from 8dbf13b to f362170 Compare September 6, 2023 11:36

amotl added 4 commits October 28, 2023 14:31

[Python/SQLAlchemy] Add example programs with SQLAlchemy and pandas

f0eae90

[Python/SQLAlchemy] Use efficient bulk insert method with pandas

411c5bd

[Python/SQLAlchemy] Add example program with Dask

0100a80

[Python/SQLAlchemy] Use efficient bulk insert method with Dask

cf10cb9

amotl force-pushed the amo/python-sqlalchemy branch from f362170 to 034c230 Compare October 28, 2023 15:22

amotl commented Oct 28, 2023

View reviewed changes

Tools: Add ngr test runner

81dac6d

amotl mentioned this pull request Oct 28, 2023

Backlog for Python #95

Open

amotl requested review from seut and matriv October 28, 2023 16:00

amotl commented Oct 28, 2023

View reviewed changes

amotl requested a review from hlcianfagna October 28, 2023 16:06

amotl force-pushed the amo/python-sqlalchemy branch from aecd5a7 to aa2551e Compare October 28, 2023 16:12

amotl marked this pull request as ready for review October 28, 2023 16:14

amotl commented Oct 28, 2023

View reviewed changes

amotl mentioned this pull request Oct 28, 2023

[Tooling] Use universal test runner ngr everywhere #96

Merged

[Python/SQLAlchemy] Add software tests and CI configuration

a43cda7

amotl force-pushed the amo/python-sqlalchemy branch from aa2551e to 7b6be1b Compare October 30, 2023 13:00

[Python/SQLAlchemy] Improve README

7355708

amotl force-pushed the amo/python-sqlalchemy branch from 7b6be1b to 7355708 Compare October 30, 2023 13:02

[Python/SQLAlchemy] Adjust dependencies

424b862

seut approved these changes Oct 30, 2023

View reviewed changes

[Python/SQLAlchemy] Use click.Choice for decoding --mode option

d55d630

amotl merged commit af6471d into main Oct 30, 2023
1 check passed

amotl deleted the amo/python-sqlalchemy branch October 30, 2023 15:47

amotl mentioned this pull request Nov 7, 2023

Add import_csv_pandas and import_csv_dask utility primitives crate-workbench/cratedb-toolkit#75

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Python/SQLAlchemy] Add example programs about efficient bulk inserts with SQLAlchemy, pandas, and Dask #64

[Python/SQLAlchemy] Add example programs about efficient bulk inserts with SQLAlchemy, pandas, and Dask #64

amotl commented May 11, 2023 •

edited

amotl Oct 28, 2023 •

edited

amotl Oct 28, 2023 •

edited

amotl Oct 30, 2023

amotl Oct 28, 2023 •

edited

amotl Oct 30, 2023

amotl Oct 28, 2023

amotl Oct 30, 2023 •

edited

amotl Oct 28, 2023

amotl Oct 30, 2023

amotl Oct 28, 2023 •

edited

amotl Oct 30, 2023

amotl Oct 28, 2023

amotl Oct 30, 2023

seut Oct 30, 2023

seut Oct 30, 2023

amotl Oct 30, 2023 •

edited

seut Oct 30, 2023

amotl Oct 30, 2023 •

edited

seut Oct 30, 2023

amotl Oct 30, 2023

		# Connect to CrateDB Cloud.
		time python insert_pandas.py --dburi='crate://admin:<PASSWORD>@example.aks1.westeurope.azure.cratedb.net:4200?ssl=true'

		source .venv/bin/activate
		pip install --upgrade --requirement requirements.txt


		logger = logging.getLogger(__name__)

		pkg_resources.require("sqlalchemy>=2.0")


		To start a CrateDB instance on your machine for evaluation purposes, invoke::

		docker run -it --rm --publish=4200:4200 --publish=5432:5432 crate

	@click.option("--mode", type=str, default="bulk", required=False, help="Insert mode. Choose one of basic, multi, bulk.")
	@click.option("--mode", type=click.Choice(['basic', 'multi', 'bulk']), default="bulk", required=False, case_sensitive=False, help="Insert mode. Choose one of basic, multi, bulk.")

[Python/SQLAlchemy] Add example programs about efficient bulk inserts with SQLAlchemy, pandas, and Dask #64

[Python/SQLAlchemy] Add example programs about efficient bulk inserts with SQLAlchemy, pandas, and Dask #64

Conversation

amotl commented May 11, 2023 • edited

Backlog

amotl Oct 28, 2023 • edited

Choose a reason for hiding this comment

amotl Oct 28, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amotl Oct 28, 2023 • edited

Choose a reason for hiding this comment

Improve documentation I

Footnotes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Improve documentation II

amotl Oct 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amotl Oct 28, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amotl Oct 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amotl Oct 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amotl commented May 11, 2023 •

edited

amotl Oct 28, 2023 •

edited

amotl Oct 28, 2023 •

edited

amotl Oct 28, 2023 •

edited

amotl Oct 30, 2023 •

edited

amotl Oct 28, 2023 •

edited

amotl Oct 30, 2023 •

edited

amotl Oct 30, 2023 •

edited