Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replication POC and tests #129

Merged
merged 49 commits into from
Feb 16, 2024
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
c461a9c
Checkpoint for replication code
zachmu Feb 7, 2024
c9702a2
More complete testing setup
zachmu Feb 7, 2024
be058eb
Updated events api
zachmu Feb 8, 2024
8fc38c2
Fix NPE
zachmu Feb 8, 2024
8a74bda
Bug fixes
zachmu Feb 8, 2024
be8271c
More testing of re-connect, still not working exactly correctly. Fill…
zachmu Feb 9, 2024
28fd3b5
Better literal quoting
zachmu Feb 12, 2024
716173d
Better update handling
zachmu Feb 12, 2024
5c84784
Better delete
zachmu Feb 12, 2024
14e4ec4
Actually replicating queries
zachmu Feb 13, 2024
943fed6
Got rid of limited listener, no longer actually used
zachmu Feb 13, 2024
9021563
Support for public schema
zachmu Feb 13, 2024
b1256fb
First fully working end to end test
zachmu Feb 13, 2024
35e1fa6
More tests, transaction framework
zachmu Feb 14, 2024
45510ae
Replication stop method, not quite working
zachmu Feb 14, 2024
2c68506
Better shutdown logic
zachmu Feb 14, 2024
b01a024
Better shutdown logic
zachmu Feb 14, 2024
c75e742
Full (simple) concurrency test
zachmu Feb 14, 2024
0e4d57e
Merge main
zachmu Feb 14, 2024
b622de1
Re-add pgloglrepl
zachmu Feb 14, 2024
03739eb
UUID test
zachmu Feb 14, 2024
e41feb5
Formatting
zachmu Feb 15, 2024
e08ef18
Fixed context leak
zachmu Feb 15, 2024
b3b788f
Adding a postgres dependency to CI
zachmu Feb 15, 2024
0ebbf37
Formatting
zachmu Feb 15, 2024
f9fda83
Vet errors
zachmu Feb 15, 2024
c1824e7
PR feedback
zachmu Feb 15, 2024
3ea3903
Killed ReplicationTestAssertion
zachmu Feb 15, 2024
a3809c3
Comments and documentation
zachmu Feb 15, 2024
2894084
Broke up replication test into its own workflow
zachmu Feb 15, 2024
f3edda8
Formatting
zachmu Feb 15, 2024
a05956e
vet
zachmu Feb 15, 2024
7ad3f61
Formatting
zachmu Feb 15, 2024
79ddade
Fixed race / hang in Stop()
zachmu Feb 15, 2024
e66725b
Set the correct WAL level in the postgres init script
zachmu Feb 15, 2024
3e0f5e3
Formatting
zachmu Feb 15, 2024
afe4aea
Trying to get args to parse correctly
zachmu Feb 15, 2024
fbd6997
Another try
zachmu Feb 15, 2024
a2ad1f7
Another try
zachmu Feb 15, 2024
df9d575
Fixed SSL test
zachmu Feb 16, 2024
cdf3a50
Another try
zachmu Feb 16, 2024
35e931d
Another try
zachmu Feb 16, 2024
5c5f67c
Formatting
zachmu Feb 16, 2024
7971c89
Formatting
zachmu Feb 16, 2024
527fa70
Trying a different image
zachmu Feb 16, 2024
c63a39f
Trying more retries
zachmu Feb 16, 2024
f11c536
Correctly skip on other platforms
zachmu Feb 16, 2024
414c1a9
Reverting retries
zachmu Feb 16, 2024
d23857f
Removed replication test in favor of an action
zachmu Feb 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,19 @@ jobs:
matrix:
platform: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.platform }}
services:
zachmu marked this conversation as resolved.
Show resolved Hide resolved
postgres:
image: postgres:15
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: password
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
steps:
- name: Setup Go 1.x
uses: actions/setup-go@v3
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ require (
github.com/golang/geo v0.0.0-20200730024412-e86565bf3f35
github.com/google/go-cmp v0.6.0
github.com/grpc-ecosystem/grpc-gateway v1.16.0
github.com/jackc/pglogrepl v0.0.0-20231111135425-1627ab1b5780
github.com/jackc/pgx/v4 v4.18.1
github.com/jackc/pgx/v5 v5.4.3
github.com/lib/pq v1.10.9
github.com/madflojo/testcerts v1.1.1
github.com/pierrre/geohash v1.0.0
github.com/sergi/go-diff v1.1.0
github.com/shopspring/decimal v1.2.0
Expand Down
7 changes: 5 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -494,6 +494,8 @@ github.com/jackc/pgconn v1.14.0 h1:vrbA9Ud87g6JdFWkHTJXppVce58qPIdP7N8y0Ml/A7Q=
github.com/jackc/pgconn v1.14.0/go.mod h1:9mBNlny0UvkgJdCDvdVHYSjI+8tD2rnKK69Wz8ti++E=
github.com/jackc/pgio v1.0.0 h1:g12B9UwVnzGhueNavwioyEEpAmqMe1E/BN9ES+8ovkE=
github.com/jackc/pgio v1.0.0/go.mod h1:oP+2QK2wFfUWgr+gxjoBH9KGBb31Eio69xUb0w5bYf8=
github.com/jackc/pglogrepl v0.0.0-20231111135425-1627ab1b5780 h1:pNK2AKKIRC1MMMvpa6UiNtdtOebpiIloX7q2JZDkfsk=
github.com/jackc/pglogrepl v0.0.0-20231111135425-1627ab1b5780/go.mod h1:Y1HIk+uK2wXiU8vuvQh0GaSzVh+MXFn2kfKBMpn6CZg=
github.com/jackc/pgmock v0.0.0-20190831213851-13a1b77aafa2/go.mod h1:fGZlG77KXmcq05nJLRkk0+p82V8B8Dw8KN2/V9c/OAE=
github.com/jackc/pgmock v0.0.0-20201204152224-4fe30f7445fd/go.mod h1:hrBW0Enj2AZTNpt/7Y5rr2xe/9Mn757Wtb2xeBzPv2c=
github.com/jackc/pgmock v0.0.0-20210724152146-4ad1a8207f65 h1:DadwsjnMwFjfWc9y5Wi/+Zz7xoE5ALHsRQlOctkOiHc=
Expand Down Expand Up @@ -524,12 +526,14 @@ github.com/jackc/pgx/v4 v4.0.0-pre1.0.20190824185557-6972a5742186/go.mod h1:X+GQ
github.com/jackc/pgx/v4 v4.12.1-0.20210724153913-640aa07df17c/go.mod h1:1QD0+tgSXP7iUjYm9C1NxKhny7lq6ee99u/z+IHFcgs=
github.com/jackc/pgx/v4 v4.18.1 h1:YP7G1KABtKpB5IHrO9vYwSrCOhs7p3uqhvhhQBptya0=
github.com/jackc/pgx/v4 v4.18.1/go.mod h1:FydWkUyadDmdNH/mHnGob881GawxeEm7TcMCzkb+qQE=
github.com/jackc/pgx/v5 v5.0.3/go.mod h1:JBbvW3Hdw77jKl9uJrEDATUZIFM2VFPzRq4RWIhkF4o=
github.com/jackc/pgx/v5 v5.4.3 h1:cxFyXhxlvAifxnkKKdlxv8XqUf59tDlYjnV5YYfsJJY=
github.com/jackc/pgx/v5 v5.4.3/go.mod h1:Ig06C2Vu0t5qXC60W8sqIthScaEnFvojjj9dSljmHRA=
github.com/jackc/puddle v0.0.0-20190413234325-e4ced69a3a2b/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v0.0.0-20190608224051-11cab39313c9/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.3/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.3.0/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle/v2 v2.0.0/go.mod h1:itE7ZJY8xnoo0JqJEpSMprN0f+NQkMCuEV/N9j8h0oc=
github.com/jcmturner/gofork v0.0.0-20180107083740-2aebee971930/go.mod h1:MK8+TM0La+2rjBD4jE12Kj1pCCxK7d2LK/UM3ncEo0o=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jmespath/go-jmespath v0.3.0/go.mod h1:9QtRXoHjLGCJ5IBSaohpXITPlowMeeYCZ7fLUTSywik=
Expand Down Expand Up @@ -614,8 +618,6 @@ github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20190605223551-b
github.com/lightstep/lightstep-tracer-go v0.18.1/go.mod h1:jlF1pusYV4pidLvZ+XD0UBX0ZE6WURAspgAczcDHrL4=
github.com/lyft/protoc-gen-star v0.5.2/go.mod h1:9toiA3cC7z5uVbODF7kEQ91Xn7XNFkVUl+SrEe+ZORU=
github.com/lyft/protoc-gen-validate v0.0.13/go.mod h1:XbGvPuh87YZc5TdIa2/I4pLk0QoUACkjt2znoq26NVQ=
github.com/madflojo/testcerts v1.1.1 h1:YsSHWV79nMNZK0mJtwXjKoYHjJEbLPFefR8TxmmWupY=
github.com/madflojo/testcerts v1.1.1/go.mod h1:MW8sh39gLnkKh4K0Nc55AyHEDl9l/FBLDUsQhpmkuo0=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
Expand Down Expand Up @@ -970,6 +972,7 @@ golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e/go.mod h1:GvvjBRRGRdwPK5y
golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220314234659-1baeb1ce4c0b/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.0.0-20220829220503-c86fa9a7ed90/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.18.0 h1:PGVlW0xEltQnzFZ55hkuX5+KLyrMYhHld1YHO4AKcdc=
golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1mg=
Expand Down
16 changes: 15 additions & 1 deletion server/ast/table_name.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,23 @@ func nodeTableName(node *tree.TableName) (vitess.TableName, error) {
if node == nil {
return vitess.TableName{}, nil
}
if node.ExplicitCatalog || node.ExplicitSchema && strings.ToLower(string(node.SchemaName)) != "information_schema" {

if node.ExplicitCatalog || node.ExplicitSchema {
if strings.ToLower(string(node.SchemaName)) == "information_schema" {
return vitess.TableName{
Name: vitess.NewTableIdent(string(node.ObjectName)),
Qualifier: vitess.NewTableIdent(string(node.SchemaName)),
}, nil
} else if !node.ExplicitCatalog && node.ExplicitSchema && strings.ToLower(string(node.SchemaName)) == "public" {
// the "public" schema is the default schema in PostgreSQL, so treat it as if it were not explicitly specified
return vitess.TableName{
Name: vitess.NewTableIdent(string(node.ObjectName)),
Qualifier: vitess.NewTableIdent(""),
}, nil
}
return vitess.TableName{}, fmt.Errorf("referencing items outside the schema or database is not yet supported")
}

return vitess.TableName{
Name: vitess.NewTableIdent(string(node.ObjectName)),
Qualifier: vitess.NewTableIdent(string(node.SchemaName)),
Expand Down
96 changes: 0 additions & 96 deletions server/limited_listener.go

This file was deleted.

Loading
Loading