Skip to content

Releases: xo/dbimp

v0.6.1

Choose a tag to compare

@kenshaw kenshaw released this 29 Sep 12:50

This release fixes two faults of the driver databend of v0.6.0, which usql found and measured. No other driver changes.

What changes for a caller

  • Databend. A connection keeps its session for as long as it lives, so USE and SET stay in effect when database/sql hands the connection to the next statement, as a pooled connection of MySQL does. In v0.6.0, the driver reset the session each time the connection went back to the pool, so a caller that runs each statement through *sql.DB, as usql does, lost a USE at once. The driver no longer implements driver.SessionResetter (D126). A caller who wants the session of the DSN again opens a new connection.
  • Databend. After a transaction fails on the server, Commit and Rollback now send the ROLLBACK that ends it. In v0.6.0, the driver refused its own ROLLBACK, and the server kept the transaction aborted until the connection closed.

Other changes

  • TDengine gets no driver here. Its REST interface ends a result that failed as valid JSON with code 0, so a driver cannot tell a cut result from a whole one (D127). docs/TDENGINE.md holds the measurements. Apache Pinot is the next target.

v0.6.0

Choose a tag to compare

@kenshaw kenshaw released this 29 Sep 12:23

This release adds the driver databend, for Databend and its SQL. It also gives every driver the same options for one statement, and it adds the key auth to the drivers influxdb and neo4j. A caller of couchbase.Option needs no change.

The Databend driver

Import github.com/xo/dbimp/databend. It registers the name databend, and it takes a DSN of this form:

databend://user:password@host:port/database?key=value

The path is the database, and default without one. The port is 8000 without one, with tls=true too. The keys are tls, auth=basic|bearer, cancel=kill|none and timezone, and the driver refuses any other key, the keys sslmode, tenant and warehouse of databend-go too (D117).

These are the rules that a caller sees:

  • The server binds each argument. A positional argument fills each ? in order, and sql.Named("k", v) fills :k. One statement cannot take both kinds. A decimal goes as a string, which keeps every digit, and a []byte is refused (D120 and D124).
  • Every value arrives as text, and the driver decodes it by the type of its column: int64, float64, *apd.Decimal, bool, []byte, time.Time and string. An Array, a Map and a Tuple arrive as text of SQL, and the driver decodes them into []any and maps. A Geometry and a Geography are WKT (D118 and D119).
  • A Bitmap cannot be read, because the server writes none of its bytes in JSON. Reading one fails with dbimp.ErrNotSupported (D119).
  • Each result arrives one page at a time. When the context ends, or the rows close early, the driver kills the query on the server, because the server runs it on when the client leaves (D123).
  • BeginTx begins a transaction, which the session of the connection carries. A DDL statement or any error ends it, and Commit then returns why. USE and SET stay on the connection until database/sql hands it to another caller (D121 and D122).
  • RowsAffected is the count that the result names, such as number of rows inserted. A statement whose result names none, such as REPLACE INTO, returns dbimp.ErrNotSupported (D125).

The tests passed against 1.2.881 and 1.2.948 as the administrator and as the ordinary user. docs/DATABEND.md holds what the driver knows about the server, and D117 to D125 in docs/decisions/ hold the reasons.

What changes for a caller

  • All drivers. Each driver takes WithTimeout, WithReadonly, WithParameter and WithDatabase, through WithOptions or as an argument of a statement, and an option for each key of its DSN that can change for one statement. An option that the server cannot honor fails the statement with dbimp.ErrNotSupported, and a value that the DSN would refuse fails it with dbimp.ErrInvalidValue. WithParameter replaces a key that the driver sets itself (D109).
  • Couchbase. couchbase.Option is now dbimp.Option[...], which stays source compatible. WithDatabase sets query_context, as WithQueryContext does.
  • Neo4j. WithTimeout sends maxExecutionTime in whole seconds, and fails with dbimp.ErrNotSupported on a release before 2026.04, which ignores it. WithDatabase and WithCancel are new. The key auth=bearer sends the password as a Bearer token (D116).
  • InfluxDB. WithDatabase, WithRetentionPolicy, WithChunked and WithDescribe are new. WithTimeout and WithReadonly fail with dbimp.ErrNotSupported, because no release has either for one request. The key auth=bearer sends the password as a token: Token to InfluxDB 2, and Bearer to 1 and 3 (D116).
  • ArangoDB. WithTimeout sends maxRuntime, and WithDatabase, WithBatch and WithCancel are new. WithReadonly holds only in a read-only transaction.
  • SurrealDB. WithDatabase and WithNamespace set the database and the namespace of one statement. WithTimeout and WithReadonly fail with dbimp.ErrNotSupported.

Other changes

  • The root package holds the options of every driver: Option, WithOptions, Resolve, Unsupported and MarshalParams. It also holds SetAuth, which sends the secret of a DSN as the key auth says.
  • The recorder of step 6 follows each page of a query, and escapes a captured value inside a JSON body.
  • TestEveryDriverTakesTheCommonOptions holds the options of each driver.

v0.5.0

Choose a tag to compare

@kenshaw kenshaw released this 29 Sep 00:36

This release adds the driver arangodb, for ArangoDB 3.12 and its language AQL. It also changes the drivers couchbase, surrealdb, neo4j and influxdb of v0.4.0, and a caller of one of them must read "What changes for a caller" below.

The ArangoDB driver

Import github.com/xo/dbimp/arangodb. It registers the name arangodb, and it takes a DSN of this form:

arangodb://user:password@host:port/database?key=value

The path is the database, and _system without one. The port is 8529 without one. The keys are tls, cancel=tag|none, batch, the size of each batch, and auth=basic|bearer. The secret is always the password of the URL: basic sends it with the user, and bearer sends it as a Bearer token, such as a JWT.

These are the rules that a caller sees:

  • A stored document or edge is one column that holds a map. Any other object gives its keys as the columns, and a scalar is one column (D89).
  • sql.Named("k", v) fills @k, and a positional argument n fills @n. A named argument that the query uses as @@k names a collection. A string or a comment that holds @@k does not count (D104).
  • Each query streams, one batch at a time. When the context ends, or the rows close early, the driver deletes the cursor, or kills the query by a comment that it adds to the query (D90 and D99).
  • BeginTx begins a stream transaction that names every collection of the database for write, except the system collections, because ArangoDB needs them before the transaction begins (D91).
  • The driver takes CREATE and DROP of collections and indexes, which AQL does not have (D92). A geo index on one field reads GeoJSON (D106).

The tests passed against 3.12.12 as the administrator and as the ordinary user. docs/ARANGODB.md holds what the driver knows about the server, and D89 to D108 in docs/decisions/ hold the reasons.

What changes for a caller

  • Neo4j. The comment of cancel=tag now ends each statement, on a line of its own, so the position of an error is where the caller wrote it (D95). A rollback now reaches the server after the context of BeginTx ends, so the server frees the locks of the transaction at once (D100). When the rows close before their end, the driver stops a statement that still runs on the server, and in a transaction it reads the rest of the answer, so that the transaction goes on (D105).
  • InfluxDB. The first column of an InfluxQL result set is measurement, and not name, so SHOW DATABASES gives measurement | name (D96).
  • All drivers. dbimp.ErrIncomplete now means that at least one row of a result set reached the caller before it failed. A failure before the first row, such as a syntax error of Couchbase, no longer wraps it (D107).
  • SurrealDB. BeginTx returns dbimp.ErrNotSupported for every option, and not only for the default ones (D54).

Other changes

  • Step 17a of docs/DRIVER.md compares each driver with couchbase before its commit (D97). Two reviews of the whole repository fixed the documents, the comments and the tests that no longer agreed with the code.
  • Each wire format meets its driver at the row, with a concrete decoder for each format (D108).
  • The parser of placeholders in the root package knows the // comment of AQL, @@name, and a name that starts with a digit.

v0.4.0

Choose a tag to compare

@kenshaw kenshaw released this 28 Sep 19:10

This release adds the driver influxdb, for InfluxDB 1, InfluxDB 2, and InfluxDB 3 and later. The drivers couchbase, surrealdb and neo4j do not change.

The InfluxDB driver

Import github.com/xo/dbimp/influxdb. It registers the name influxdb, and it takes a DSN of this form:

influxdb://user:password@host:port/database?key=value

The path is the database. A token is the password, sent with basic authentication. Without a port, the driver uses 8086 for version=1 or version=2, and 8181 otherwise.

The driver speaks two dialects:

  • influxdb is SQL, on InfluxDB 3 and later.
  • influxql is InfluxQL, on every release.

The key sqlmode chooses the dialect, as sslmode does for PostgreSQL. Its default, prefer, asks GET /ping for the release, and speaks SQL to InfluxDB 3 and InfluxQL to InfluxDB 1 and 2. With disable or allow, the driver sends no ping, and the key version names the release. Inside sql.Conn.Raw, influxdb.Version returns the release, and influxdb.Dialect returns the dialect of the connection.

These are the rules that a caller sees:

  • SQL learns the columns and their types from DESCRIBE, because the JSON of the server leaves out every NULL. The key describe=disable turns that off. A NaN or an infinity from an expression reads as math.NaN(), because the JSON writes all three alike.
  • InfluxQL gives each series its own result set. Its first columns are name and the tags of the series. Call Rows.NextResultSet to read the next one. A statement with no series is a result set with no columns.
  • The column time of InfluxQL is a time.Time. A number in InfluxQL is an int64, a uint64 above the range of int64, or a float64, by its text, so a whole float reads as an int64.
  • INSERT [INTO <database>[.<retention-policy>]] <line protocol> writes line protocol, as the influx shell does, in both dialects. Each $name or $1 argument becomes a literal of line protocol. A NULL leaves its tag, its field or its timestamp out of the line.
  • DELETE goes to the server. InfluxDB 1 and 2 run it. InfluxDB 3 Core has no delete of points, and refuses it.
  • The driver asks InfluxDB 1 for chunks, so that a large result streams. The key chunked=disable turns that off.
  • InfluxDB has no transactions, so BeginTx returns dbimp.ErrNotSupported.

The tests passed against 1.11.8, 1.13.1, 2.8.0, 2.9.1, 3.9.13, 3.10.6 and 3.11.5, as the administrator, and as the ordinary user on InfluxDB 1 and 2. docs/INFLUXDB.md holds what the driver knows about each release, and decisions D78 to D87 in docs/decisions/ hold the reasons.

For the authors of a driver

  • dbimptest.RoundTripCase has three new fields: Column, the column of the value in the select; SkipUpdate, the reason why an update cannot run; and an empty Delete, for a database that cannot delete one row (D86). Each keeps the old behavior when it is not set.
  • The recorder in dbimptest/cmd/record keeps a body that the server closed before its end, and Replay closes it at the same place. A request can name the releases that it runs on, and -ordinary can be empty for a release that has no ordinary user.

Other changes

  • The list of targets drops Gel, Blazegraph and Stargate, because each one is retired (D84). Milvus and PostgREST move to P2 (D87).
  • A SurrealDB RecordID writes its text form in JSON (D70). The code was in v0.3.0, and this release accepts the decision.

v0.1.0

Choose a tag to compare

@kenshaw kenshaw released this 27 Sep 09:34

The first release of dbimp. It holds one driver, for Couchbase, and the code that every later driver shares.

The Couchbase driver

github.com/xo/dbimp/couchbase is a database/sql driver for the Couchbase query service and SQL++. It replaces github.com/couchbase/go_n1ql and xo/n1ql. It uses the Go standard library and github.com/cockroachdb/apd/v3, and it needs no cgo.

import (
	"database/sql"

	_ "github.com/xo/dbimp/couchbase"
)

db, err := sql.Open("couchbase", "couchbase://user:pass@localhost:8093/")
  • It registers one name, couchbase, and takes only a couchbase:// URL. The keys of the URL are tls, query_context, scan_consistency, timeout, durability_level and txtimeout. An unknown key or a repeated key is an error.
  • The server binds each argument. ?, $1 and sql.Named all work.
  • A row keeps the order of the columns that the server sends, and each value is a Go value: int64, float64, string, bool, []any, map[string]any, or nil for NULL and MISSING. An integer too large for an int64 is a *apd.Decimal.
  • A string scans into a []byte as base64, and as its own bytes when it is not base64.
  • It reads a result one token at a time as it arrives, and it never holds the whole result in memory.
  • database/sql transactions map onto the transactions of the query service. On a server with one node, set durability_level=none, because the default durability cannot commit there. The server ends a transaction after 15 seconds unless txtimeout is longer.
  • WithOptions and the Option functions set the options of one statement or one transaction. WithParameter sets any parameter of the request by its name.
  • A failed response is a *couchbase.ResponseError, and errors.As finds each couchbase.Error in it, with its code.
  • It stops a query when its context ends.

It is tested on Couchbase 7.2.9, 7.6.12 and 8.0.3, as an administrator and as an ordinary user. The tests cover CRUD, each native type, and each feature in testdata/couchbase/features.json. docs/COUCHBASE.md holds every fact that was measured.

What a consumer changes

  • The scheme is couchbase, not n1ql. dburl keeps n1ql as an alias.
  • A value is a decoded Go value, not JSON text, so code that called strconv.Unquote on a value no longer needs it.
  • The columns keep the order that the server sends. Couchbase 7.2 sends them in the order of their names.
  • LastInsertId returns an error, and RowsAffected returns the count of mutations.

For the authors of drivers

  • The root package dbimp holds the HTTP client, the readers of rows, the decoders of values and the parser of placeholders.
  • dbimptest holds the helpers for the tests of a driver: recorded exchanges, the contract suite, the round trip of a type, and the survey of features.
  • docs/DRIVER.md holds every step to add a driver, and the gates in gates_test.go hold the steps.