Releases: dolthub/doltgresql
Releases · dolthub/doltgresql
0.9.0
Merged PRs
doltgresql
- 368: /.github/workflows/cd-release.yaml: fix sed command for server version
- 365: add smoke tests for perl, php and ruby
- 362: Perform function analysis once
Functions did their entire overload resolution duringEval
, meaning if a function was called 1000 times, then we did overload resolution 1000 times for the exact same function. This changes it so that we only do it once. This optimization was made while investigating our performance benchmarks.
Of note, errors found during overload resolution are stashed for later. This ensures that other steps in the pipeline retain their error priority. Previously, when this was all done inEval
, stashing wasn't needed. - 355: Added correct timeouts to server
Fixes #348
Idle connections were timing out in 60s before this change. - 350: Reworked config
This change reworks how the data dir, config file and related initialization values are loaded. The main effect is that it's now possible to rundoltgres
without a config.yaml file.
The sever config is now resolved in this order- --config flag
- config.yaml file if present
- Built-in default config values
The data-dir for this config is then overridden as necessary with a similar pattern: - --data-dir flag
- The value in a config.yaml file if, if a file was loaded
- The env var
DOLTGRES_DATA_DIR
- The default ($home/doltgres/databases)
- 343: support for drop table without a schema name
This PR has the tests. The fix for the behavior is in dolthub/dolt#7952 - 339: Release v0.8.0
Created by the Release workflow to update DoltgreSQL's version
Closed Issues
- 348: disabling the session after executing the request
0.8.0
Merged PRs
doltgresql
- 336: Added string literal to array interpretation
This adds functionality so that string literals such as'{1,2,3}'
can be interpreted as arrays of any type. This builds off of the #333 PR, which enables this to "just work" for all contexts (includes casting, inserting into columns, automatic operator resolution, etc.). This also includes additional tests for a few bugs that weren't caught in the previous PR. - 333: Removed sql.Convert usage, moved everything to casts
This removes our reliance onsql.Convert
and changed everything to make use of casts. Casts have three contexts: explicit (value::type), assignment (INSERT
andUPDATE
statements), and implicit. In addition to moving to casts, I've also added assignment casts.
This move also comes with a massive change to how function overloads are handled. Overload resolution is handled through available casts, so by removing the hacky behavior that was in place, we're now able to unskip a number of tests that just could not be handled through the previous logic. There were other tests that were also incorrect, and those have been fixed.
Overall, this has resulted in a fairly substantial decrease in code throughout the codebase, in addition to allowing for types to finally have "full" support.
Companion PR: - 325: Update get-doltgres-correctness-job-json.sh
- 320: Update schema path in scripts
- 319: Added comparisons for JSONB
This adds comparisons forjsonb
. In the original JSON PR, I mistakenly only checked ifjson
had comparisons, and that type does not.jsonb
, however, does have comparisons, so this implements those. In addition, I've added a massive collection of rows that it properly sorts over. Test results were lifted straight from a Postgres 15 server to ensure accuracy. - 317: /utils/doltgres_builder/run.go: fix doltgres builder paths
- 302: Added JSON types
This adds JSON types, and also fixes a bug where the text types would incorrectly persist an empty string as a NULL. - 288: Release v0.7.5
Created by the Release workflow to update DoltgreSQL's version - 287: Added SERIAL types along with remaining SEQUENCE functionality
Companion PRs:- dolthub/vitess#347
- dolthub/go-mysql-server#2504
This adds theSERIAL
types, along with the remaining functionality for sequences that was not present in the first PR. In addition, this adds our first uses of injected analyzer steps, and introduces custom nodes used for the sequence statements (instead of stored procedures).
- 269: New tests for schema support
Also changes expected errors to match an error string. Not all tests are passing yet. - 185: Update README.md
Closed Issues
0.7.5
Merged PRs
doltgresql
- 283: scripts/build_binaries.sh: Fix regression where we accidentally shipped all linux/amd64 binaries instead of platform-specific ones.
- 276: move config to its own package and update dolt servercfg package
- 273:
EXECUTE
doesn't send RowDescription
This PR fixes issue connecting to doltgres server using Postgres JDBC driver.
In Postgres docs, it says, "Execute
doesn't cause ReadyForQuery or RowDescription to be issued."
Also, queries that don't return set of rows sendNoData
in response toDescribe
message. - 266: Release v0.7.4
Created by the Release workflow to update DoltgreSQL's version - 255: only some queries send
RowDescription
message- Only queries that returns set of rows should send
RowDescription
message. This includesSELECT
,FETCH
,SHOW
, etc. - The
CommandTag
is now set by theStatementTag
from the parsed query. - Empty query should send
EmptyQueryResponse
message.
- Only queries that returns set of rows should send
- 219: Adds CREATE SEQUENCE and DROP SEQUENCE
Companion PR:- dolthub/dolt#7848
This adds functionality forCREATE SEQUENCE
andDROP SEQUENCE
. For sequences, this is what is in the PR: - MINVALUE
- MAXVALUE
- INCREMENT
- AS DATA TYPE
- START WITH
- CYCLE
This is what is missing: - TEMPORARY
- UNLOGGED
- CACHE
- OWNED BY
- SERIAL
- Merging
- Table Querying
Of what's missing, merging andSERIAL
are the main additions, withOWNED BY
required for properSERIAL
support. Table querying is fairly straightforward (thepg_sequences
system table, etc.). The others are, hopefully, unnecessary for now and can be postponed, although perhaps we can save theCACHE
value and ignore it since it's primarily a performance-oriented option.
With that,SERIAL
and merging will come in the next PR, since this is already very large. Also of note, theWriter
andReader
utilities were implemented for my Convert-to-Cast PR but they were useful here, so I pulled them from that in-flight PR and added them here.
- dolthub/dolt#7848
Closed Issues
- 240: Returned result differs from Postgres
0.7.4
0.7.1
Merged PRs
doltgresql
- 261: add the minver_validation.txt file to the release commit
Does the same thing as: https://github.com/dolthub/dolt/blob/main/.github/workflows/cd-release-pgo.yaml#L79 - 260: Release v0.7.0
Created by the Release workflow to update DoltgreSQL's version
Closed Issues
0.7.0
Merged PRs
doltgresql
- 259: build parser earlier in release process
The parser needs to be built beforego run -mod=readonly ./utils/genminver_validation/ $FILE
- 258: bump go version to 1.22.1
- 253: /.github/workflows/bump-dependency.yaml: use personal tokens to open prs to ensure ci runs
- 245: Fixes LIMIT
Fixes #172 - 244: Fixed replication tests
- 232: Added Doltgres-implementation of RootValue
Companion:- dolthub/dolt#7829
This is Doltgres' implementation ofRootValue
. For now, it's essentially taken wholesale from Dolt as a starting point. Of note, we probably want to at least get something different for our collation values sometime soon. Also, we're using the same serial identifier for root values between Doltgres and Dolt. This simplifies a good bit of code in Dolt, so taking on the naming requirement seems well worth it.
- dolthub/dolt#7829
- 231: kill subcommands
- 226: Add better cast tests for
name
, fix bool -> name cast - 221: move main
Move the main package to cmd/doltgres so that the source built binary matches the released binary. - 214: scripts/build_binaries.sh: Build releases with CGO enabled.
- 210: Implement
xid
type - 204: Implement oid type
- 202: Implement
name
type - 191: Fix typo and go test command in docs
- 188: Added website and documentation site to README
- 184: /.github/markdown-templates/dep-bump.md: add md template
- 183: use personal token so release pr runs CI
- 182: Added explicit initialization order to packages
This replaces all of theinit()
functions throughout the project with a package-levelInit()
function that is called from a newinitialization
package. This now gives an explicit ordering to the calls. The biggest disadvantage of this approach is that some packages cannot have package-level tests that can access package-private variables without potentially causing import cycles if they need to initialize another package. This was also a limitation with the previousinit()
approach as well, but the vast majority of our tests are done from thetesting
subdirectories (in part due to this limitation), so we've not lost anything by changing to this approach. - 181: Support for schemas
Things that work:Create schema
- Explicit schema name qualifiers in statements, with and without database name qualifiers
Things that don't work: - Foreign keys
- Search path support for resolving table names
- Automatically creating a
public
schema for new databases - Revision name qualifiers (branch name DBs)
- 180: Update bump-dependency.yaml
- 179: Update bump-dependency.yaml
- 178: Revert "/.github/workflows/cd-release.yaml: revert bump of dolthub actions"
Reverts #176 as these actions have been updated and work now - 177: Release v0.6.0
Created by the Release workflow to update DoltgreSQL's version - 168: support
CREATE VIEW
Depends on:
Closed Issues
- 172: The LIMIT expression is not working
0.6.0
Merged PRs
doltgresql
- 176: /.github/workflows/cd-release.yaml: revert bump of dolthub actions
- 174: Db/bump actions
- 169: doltgresql version --dolt
- 167: Added more types, reworked cast framework
This adds several new types, and also reworks type serialization and casting, among a few other things. This is not nearly as tested as I'd want it to be, as I discovered that the implicit/explicit casting is fundamentally incompatible with how we usesql.Type.Convert
, so that entire system needs to be "removed" for Doltgres (that is, it functionally should not do anything).
In addition, in an effort to be a little forward-thinking since I'm assuming we'll be breaking things a bit once the conversion system is removed, I added a form of type versions so that we can update types as we discover deficiencies. For example, let's say a type uses-1
to mean "no limit", and then we later find out that-1
is a valid value. We can make that distinction using type versions. In addition, this will break all existing databases due to the type serialization change, but I think it's fine as all currently known users should be able to rebuild the database since their replicas. - 166: add parsing support for
OUT
andINOUT
argmode for routines
Aggregate only allowsIN
andVARIADIC
argument modes. Routines including functions and procedures allowsIN
,VARIADIC
,OUT
andINOUT
modes. The parser will allowOUT
andINOUT
for aggregates, soTODO
there needs a check during ast evaluation/conversion for these modes in aggregate statements.- small fix for supporting
SETOF
modifier option forCREATE FUNCTION
statement. - small fix for supporting
db.aggregate_name
syntax forAGGREGATE
statements.
- small fix for supporting
- 165: Removed inline VarChar implementation
This removes the inline VarChar variable. It's not used anywhere, so it's best to remove it before we use it in a location that we don't mean to. We still retain the inline characteristics for user-defined columns that define a limit below the inline max though. - 164: Reworked array serialization format
This is building off of the following PR comment:
#162 (comment)
In Postgres, there seems to be a field size limit of 1GB, so I've used that to determine that we can just useuint32
s everywhere, rather than having two formats withuint16
anduint64
(which were attempts at conserving space). - 163: use auto_increment for SERIAL type
This PR reverts the latest change on using auto_increment for SERIAL type column definition.
Note: INSERT on SERIAL type as primary key does not work for now. - 162: All Array Types
This PR adds array types for all existing types. This is done by providing a wrapper that handles all of the array-functionality over the base type. In addition, this removes the mock array functionality that only worked to demonstrate the viability ofboolean[]
, replacing the functionality with fully working equivalents. - 161: remove usage of auto_increment for SERIAL type for column definition
- 159: small fix for
SET
syntax accepting multiple values as a single value - 158: support
COMMENT
statement syntax - 157: Release v0.5.0
Created by the Release workflow to update DoltgreSQL's version - 156: Use go 1.22 for release process
- 152: support
AGGREGATE
statement syntaxes
This PR adds support forCREATE AGGREGATE
andDROP AGGREGATE
statements syntaxes.
Closed Issues
0.5.0
Merged PRs
doltgresql
- 152: support
AGGREGATE
statement syntaxes
This PR adds support forCREATE AGGREGATE
andDROP AGGREGATE
statements syntaxes. - 151: /{go.mod,.github}: bump go version
- 150: support
TYPE
andDOMAIN
statements syntaxes
This PR adds support forCREATE TYPE
,ALTER TYPE
,CREATE DOMAIN
,ALTER DOMAIN
ANDDROP DOMAIN
statement syntaxes. - 149: support
LANGUAGE
syntaxes
This PR adds support forCREATE
,ALTER
andDROP
LANGUAGE
statements syntaxes. - 148: Replication options for config.yaml
Also fixed an issue where the --config option wasn't working (looking in the wrong directory). - 147: Added unary and binary operator functions
This adds all of the internal functions for unary and binary operators (for types that are currently implemented). - 145: support all
SEQUENCE
statements syntax
This PR adds complete support for parsingSEQUENCE
syntaxes. - 144: INT4 ORDER BY Workaround
There are two bugs here. The first is thatINTEGER
andINT
were parsing asINT8
. The second is thatORDER BY 1
was failing for tables that had anINT4
but not anINT8
. For the first, our parser simply used the wrong bit width, which is fairly straightforward.
The second bug was far trickier to track down. Essentially,ORDER BY 1
has been broken, however some combinations of tables and rows would result in the correct order on accident, and that's what led to the confusion on how the second bug worked. It's even more coincidental that the literal1
inORDER BY 1
happened to correspond to the first value in the affected tests, further masking the issue.
Fundamentally, the issue has not been fixed, and this submits a workaround. The root of the issue is that GMS expects thevitess.SQLVal
type, and does not handle our case (Postgres literals). As a workaround, we're converting our Postgres literals tovitess.SQLVal
specifically for this exact case. The real fix would be for GMS to defer to Doltgres when encountering types that it cannot hardcode (or to un-hardcode types, which seems relatively bad for GMS just to work better with Doltgres). We do not have a standard way of implementing this differing interface yet, so we'll punt on that and just get this workaround in. - 143: Support configuration parameters handling
This PR includes changes for:- The complete list of configuration parameters is in.
- All of the postgres configuration parameters will be added to existing system variables list in GMS. There are two parameters that have the same name as mysql system variables, which will be overwritten by the postgres parameter. This is temporary until we completely separate the use of them.
- Current implementation of
sql.SystemVariableRegistry
interface is for Dolt only, but we add all the postgres configuration parameters to theSystemVariables
global variable in GMS. This should be updated to handle Doltgres config parameters separately in future. - The default and only supported scope of postgres parameters is
SESSION
, which is used asSESSION
scope in GMS. There is no GLOBAL scope inpostgres
to be set during a session.
Depends on GMS PR: dolthub/go-mysql-server#2375
Depends on Dolt PR: dolthub/dolt#7579
Next up: - need more support for
Parameter
such as accept setting memory and time values that has unit information, etc. - need privileges and role checking for specific parameters.
- support setting parameter defaults from configuration file.
- `LOCAL scope support.
- 142: Added a plethora of function tests
This adds tests for most of the functions, along with the generation code that grabs results from a running PostgreSQL 15 to create tests with. - 141: added nil check in
Addition
expression- new Addition expression can be used without children expressions defined
Note: majority of the tests regressed because of not being able to check doltgres-specific types in GMS.
E.g: doltgrestypes.Int64Type
is not recognized asNumberTypeImpl_
in GMS.
- new Addition expression can be used without children expressions defined
- 140: Tests and fixes for more replication scenarios
- 139: add
TRIGGER
statements parsing
Adds syntax parsing support for:CREATE TRIGGER
ALTER TRIGGER
DROP TRIGGER
- 138: add
VIEW
andMATERIALIZED VIEW
statements parsing
Adds syntax parsing support for:CREATE VIEW
CREATE MATERIALIZED VIEW
ALTER VIEW
ALTER MATERIALIZED VIEW
- 137: More replication tests
Tests discovered and fixed several issues from first draft, mostly involving stopping and resuming replication and many races in testing and the replicator itself.
Next step: error handling tests using toxiproxy
Final step: UX and docs - 136: support
PROCEDURE
statements syntax parsing
This PR includes syntax parsing support for:CREATE PROCEDURE
DROP PROCEDURE
ALTER PROCEDURE
ALTER FUNCTION
- 135: support parsing
ALTER INDEX
stmt
This PR also fixes all the synopsis that had[ $argmode$ ] [ $argname$ ] $argtype$ [ , ... ]
to correct format of{ [ $argmode$ ] [ $argname$ ] $argtype$ } [ , ... ]
. The tests that are affected by this change is updated. - 134: Update update-perf.sh
- 133: Function Overload Rework
What this adds:- Type resolution for all extended (Doltgres) types
- Proper type casting between types (extendable to support user-defined conversion)
- Additional Types
The previous function overload framework only worked in a very limited capacity. This PR intends to remove those limitations, allowing for all current (and future) Postgres functions to be defined. In general, this is also designed to support user-defined functionality, or at least designed in such a way that it will be relatively easy to fully implement it later on.
Related PR: dolthub/dolt#7537
- 132: New test runner
Test runner for docker-based replication tests - 131: Docker test for postgres replication
Seeking some feedback on this approach. This works locally for me on my local Docker when I build the image from the package root:Next step would be to create a github action that runs this docker file and a workflow trigger for that action. Is this going to work? I looked into the examples in the Dolt repo and got confused by the paths that github actions appears to use at execution time, so could use some advice on best practices.docker build --file=testing/ReplicationTestDockerfile .
- 130: support for parsing
CREATE TABLE
andALTER TABLE
statement syntax
This PR adds support for parsingCREATE TABLE
andALTER TABLE
statements
It also removes some parsing code for statements that are not supported in Postgres. - 129: replication POC and tests
Looking for feedback on this approach before polishing it off for submission.
Missing / next steps:- Keyless table tests
- Start up postgres during CI so the replication tests can run correctly there
- UX to configure and start up replication for the server
- 128: [auto-update-readme] by coffeegoddd
- 127: [auto-update-readme] by coffeegoddd
- 126: Release v0.4.0
Created by the Release workflow to update DoltgreSQL's version
Closed Issues
- 146: someone is trolling doltgresql stars and sending unsollicited emails
0.4.0
Merged PRs
doltgresql
- 125: Added UUID type
This adds the UUID type. Ignore the serialization portion for now, I'll figure out a different serialization scheme later that should work for all types (including user-defined types). While this works, we need to forward the type information through to the connector so it can use the correct OID. For now though, this should be enough to unblock. - 123: Fixed command docs generation
Sometimes, command docs would generate duplicate tests. This was due to the RNG selecting the same number multiple times. This fixes this issue, along with fixing all of the affected tests. There are still duplicate tests, but that is due to the synopses defining the duplicates, rather than the generation, which I think is fine. - 122: support syntax parsing for INDEX statements
This PR adds syntax parsing support for:CREATE INDEX
CREATE TABLE AS
It also splitspostgres/parser/sem/tree/create.go
file intocreate_*.go
files for each specific statements.
- 121: Referencing columns within arrays
This makes use of an addition to Vitess and GMS that allows for name resolution to occur for injected expressions. The newARRAY[...]
smoke test demonstrates this.
Related PRs: - 120: Added BOOLEAN and BOOLEAN[] types
This adds rough support for bothBOOLEAN
andBOOLEAN[]
types. This is primarily to show a working implementation of the newExtendedType
interface that was added to GMS to support these new types, demonstrating that we can now add standard PostgreSQL types, as well as array types (along with accompanying functionality, such asARRAY[]
support).
Related PRs: - 119: support parsing create extension and function statements
Added parsing support to following statements:CREATE EXTENSION
CREATE FUNCTION
COMMENT ON EXTENSION
DROP EXTENSION
DROP FUNCTION
- 118: [auto-update-readme] by coffeegoddd
- 117: [auto-update-readme] by coffeegoddd
- 115: Update get-doltgres-correctness-job-json.sh
- 114: /testing/logictest/harness/doltgres_server_harness.go: catch error
- 113: /README.md: changes for automated perf updates
- 112: /.github/{scripts,workflows}: add script for updating readme, fix other workflows
- 109: /.github/workflows: enable nightly correctness, add latency on pull comment
- 108: /.github/{scripts,workflows}: add correctness workflows
- 107: /.github/scripts/performance-benchmarking/get-{postgres,doltgres}-doltgres-job-json.sh: set email template
- 106: support ddl for schema and database
This PR adds missing postgres syntax for:CREATE SCHEMA
CREATE DATABASE
DROP DATABASE
- 105: /.github/workflows: skip tpcc, add nightly benchmarks and benchmarks on release
- 104: /.github/workflows/email-report.yaml: fix email report workflow
- 102: Db/fix ci
- 100: /.github/{actions,scripts,workflows}: add benchmarking workflows to ci
- 99: Better error handling
Server no longer always forcefully closes a connection when catching a panic, which is still very common because of unsupported features that panic, as well as unintended panics due to bugs.
This change also refactors the connection handling logic to extract a new ConnectionHandler type.
Also fixes a problem where a errors during a Query execution would hang a connection while it waited for a Sync message that never came. AddedwaitForSync
flag in ConnectionHandler to manage this bit of state. - 96: Updated contribution guide
- 95: Better prepared statement support
Now handles type casts for placeholders, as well as matching a larger variety of expressions.
Companion PR:
dolthub/go-mysql-server#2272 - 93: PostgreSQL function PR feedback
- 91: Added more PostgreSQL functions
This only includes quite a few functions, but doesn't include their tests. I was planning on including those too, but it's turning into a "more work than expected" type of situation, and if I'm going to solve those problems, I'd rather do it with all of the rest of the functions too, so that I can just get all the functions done for all the tests as once. This implements every function that I think we can implement for now based on our currently-supported types (without ripping out the implementation as soon as we get proper type support).
In the meantime, we can get these functions in, and get the tests in later. - 89: add utility for building doltgres binaries
- 87: Prepared statement support
This PR implements postgres prepared statements with support for$1
style parameters in query strings. If this approach is reasonable, the next step would be to add support for additional types and lots more tests.
Also missing is support for the client manually specifying the types of parameters in aParse
message, which is required for certain statements where the type information cannot be determined based on referenced columns.
Companion PRs:
dolthub/go-mysql-server#2239
dolthub/vitess#299 - 86: support postgres syntaxes - 2
This PR includes some postgres-specific syntax support.ALTER DATABASE
GRANT
REVOKE
ALTER DEFAULT PRIVILEGES
- 85: support some postgres syntaxes - 1
This PR includes some postgres-specific syntax support.ABORT
ALTER AGGREGATE
ALTER COLLATION
ALTER CONVERSION
- 84: Release v0.3.1
Created by the Release workflow to update DoltgreSQL's version - 83: starting doltgres server creates db in defined directory
Currently, doltgres uses dolt init when starting doltgres server with data dir set to empty directory. When there is no dolt or doltgres configuser.name
anduser.email
, it fails to create a new database with error message fromdolt init
, which suggests to setdolt
user info. Current fix will not return error; instead, it will use user name,postgres
and email,postgres@somewhere.com
if it's not set. - 82: Added a framework for creating PostgreSQL functions
This is the 2nd version of my attempt at implementing functions.
The first iteration hijacked*tree.FuncExpr
and special-cased names of functions that matched the built-in ones for Postgres. Then it returned a new node that approximated the functionality. In some cases this was as simple as returning a different*vitess.FuncExpr
node that pointed to a different function, however some cases did not have direct analogues in MySQL. Originally I was going to ignore those, but since I was working on functions anyway, I decided to tackle them to get it over with, similar to my approach with the entire AST conversion. Well that's when I started running into two key issues:- No straightforward conversion for some functions
- Different behavior for similar functions
The second was born from the first, as I needed to extend my tests to make sure my massive nodes actually worked for most cases. However, extending the tests showed some issues that are somewhat fundamentally different to how MySQL approaches functions, with a big one being overloading.
The first was a major issue though. I'll post an example in another comment, but some of the nodes became almost unreadable, and they also took forever to create (multiple hours per function, hence the continual delays). The only real maintainable alternative would be to skip the AST conversion pipeline and jump straight to the GMS expressions, but that posed its own issues. Not only is there a lot of boilerplate for expressions, but some of the more sophisticated aspects (such as the origin of values, which I'm calling theirSource
) are specific to PostgreSQL, So I instead decided to modify the entire set of functions by replacing all of the built-ins (Postgres doesn't want MySQL's built-ins anyway) with a custom "compiled" function structure.
So now, this mimics a bit how overloaded stored procedures work. We define a set of functions under a single name, and those functions become overloads for that name. Whenev...
0.3.1
Merged PRs
doltgresql
- 83: starting doltgres server creates db in defined directory
Currently, doltgres uses dolt init when starting doltgres server with data dir set to empty directory. When there is no dolt or doltgres configuser.name
anduser.email
, it fails to create a new database with error message fromdolt init
, which suggests to setdolt
user info. Current fix will not return error; instead, it will use user name,postgres
and email,postgres@somewhere.com
if it's not set. - 80: Release v0.3.0
Created by the Release workflow to update DoltgreSQL's version
Closed Issues
- 81: cannot run
doltgres
with data dir set to empty directory