Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unite admin-console/psql-console pre-routing logic. #259

Merged
merged 3 commits into from
Aug 7, 2023

Conversation

reshke
Copy link
Collaborator

@reshke reshke commented Aug 3, 2023

essentially fixes #255

@reshke reshke enabled auto-merge (squash) August 3, 2023 09:20
@reshke reshke requested a review from Denchick August 3, 2023 09:32
@Denchick
Copy link
Collaborator

Denchick commented Aug 4, 2023

Still crashing

router log:

➜  spqr git:(admin_console_imprv) ./spqr-router -c examples/2shardproxy.yaml run
2023/08/04 13:25:40 Running config: {
  "log_level": "log",
  "time_quantiles": null,
  "daemonize": false,
  "maintain_params": false,
  "with_jaeger": false,
  "pgproto_debug": false,
  "pid_filename": "",
  "log_filename": "",
  "host": "::1",
  "router_port": "6432",
  "admin_console_port": "7432",
  "grpc_api_port": "7010",
  "world_shard_fallback": true,
  "show_notice_messages": false,
  "init_sql": "examples/init.sql",
  "memqdb_backup_path": "memqdb.json",
  "router_mode": "PROXY",
  "jaeger_url": "",
  "frontend_rules": [
    {
      "db": "db1",
      "usr": "user1",
      "auth_rule": {
        "auth_method": "ok",
        "password": "strong"
      },
      "pool_mode": "TRANSACTION",
      "pool_discard": false,
      "pool_rollback": false,
      "pool_prepared_statement": true,
      "pool_default": false
    },
    {
      "db": "",
      "usr": "",
      "auth_rule": {
        "auth_method": "ok",
        "password": ""
      },
      "pool_mode": "TRANSACTION",
      "pool_discard": false,
      "pool_rollback": false,
      "pool_prepared_statement": false,
      "pool_default": true
    }
  ],
  "query_routing": {
    "multicast_unroutable_insert_statement": false,
    "default_route_behaviour": ""
  },
  "frontend_tls": {
    "sslmode": "disable",
    "key_file": "/etc/odyssey/ssl/server.key",
    "cert_file": "/etc/odyssey/ssl/server.crt",
    "root_cert_file": ""
  },
  "backend_rules": [
    {
      "db": "db1",
      "usr": "user1",
      "auth_rules": null,
      "auth_rule": null,
      "pool_default": false,
      "connection_limit": 100
    },
    {
      "db": "",
      "usr": "",
      "auth_rules": null,
      "auth_rule": null,
      "pool_default": true,
      "connection_limit": 2
    }
  ],
  "shards": {
    "sh1": {
      "hosts": [
        "localhost:5550"
      ],
      "type": "DATA",
      "tls": {
        "sslmode": "disable",
        "key_file": "/etc/odyssey/ssl/server.key",
        "cert_file": "/etc/odyssey/ssl/server.crt",
        "root_cert_file": ""
      }
    },
    "sh2": {
      "hosts": [
        "localhost:5551",
        "localhost:5552"
      ],
      "type": "DATA",
      "tls": {
        "sslmode": "disable",
        "key_file": "/etc/odyssey/ssl/server.key",
        "cert_file": "/etc/odyssey/ssl/server.crt",
        "root_cert_file": ""
      }
    }
  },
  "reuse_port": false
}
{"level":"info","node":"sh1","time":"2023-08-04T13:25:40+05:00","message":"adding nodee"}
{"level":"info","node":"sh2","time":"2023-08-04T13:25:40+05:00","message":"adding nodee"}
{"level":"info","path":"examples/init.sql","time":"2023-08-04T13:25:40+05:00","message":"autoconf file found"}
{"level":"info","time":"2023-08-04T13:25:40+05:00","message":"executing init sql"}
{"level":"info","query":"ADD SHARDING RULE r1 COLUMNS w_id;","time":"2023-08-04T13:25:40+05:00"}
{"level":"info","query":"ADD SHARDING RULE r2 COLUMNS id;","time":"2023-08-04T13:25:40+05:00"}
{"level":"info","query":"ADD SHARDING RULE r3 COLUMNS bid;","time":"2023-08-04T13:25:40+05:00"}
{"level":"info","query":"ADD KEY RANGE krid3 FROM 21 TO 30 ROUTE TO sh2;","time":"2023-08-04T13:25:40+05:00"}
{"level":"info","query":"ADD KEY RANGE krid2 FROM 11 TO 20 ROUTE TO sh1;","time":"2023-08-04T13:25:40+05:00"}
{"level":"info","query":"ADD KEY RANGE krid1 FROM 1 TO 10 ROUTE TO sh1;","time":"2023-08-04T13:25:40+05:00"}
{"level":"info","count":6,"filename":"examples/init.sql","time":"2023-08-04T13:25:40+05:00","message":"successfully init queries from file"}
{"level":"info","address":"[::1]:7432","time":"2023-08-04T13:25:40+05:00","message":"SPQR Administative Console is ready on"}
{"level":"info","address":"[::1]:6432","time":"2023-08-04T13:25:40+05:00","message":"SPQR Router is ready by postgresql proto"}
{"level":"info","address":"[::1]:7010","time":"2023-08-04T13:25:40+05:00","message":"SPQR GRPC API is ready on"}
{"level":"info","address":"localhost:6432","time":"2023-08-04T13:25:40+05:00","message":"SPQR Router is ready by postgresql proto"}
{"level":"info","ssl":false,"time":"2023-08-04T13:25:49+05:00","message":"init client connection"}
{"level":"info","time":"2023-08-04T13:25:49+05:00","message":"console.ProcClient start"}
{"level":"info","ssl":false,"time":"2023-08-04T13:25:58+05:00","message":"init client connection"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x139bec7]

goroutine 52 [running]:
github.com/jackc/pgx/v5/pgproto3.(*Backend).Send(0x0, {0x1b5a980, 0x212e7c0})
        /Users/denchick/Code/spqr/vendor/github.com/jackc/pgx/v5/pgproto3/backend.go:60 +0x27
github.com/pg-sharding/spqr/router/client.(*PsqlClient).Send(0xc0004bc270, {0x1b5a980?, 0x212e7c0})
        /Users/denchick/Code/spqr/router/client/client.go:759 +0xba
github.com/pg-sharding/spqr/router/console.(*Local).Serve(0xc00012ccd0?, {0x1b5ecb0, 0xc000110010}, {0x1b6a868, 0xc0004bc270})
        /Users/denchick/Code/spqr/router/console/console.go:101 +0x293
github.com/pg-sharding/spqr/router.(*InstanceImpl).serv(0xc00040cbd0, {0x1b623b0?, 0xc000012028}, 0x1)
        /Users/denchick/Code/spqr/router/instance.go:144 +0x472
github.com/pg-sharding/spqr/router.(*InstanceImpl).RunAdm.func2()
        /Users/denchick/Code/spqr/router/instance.go:243 +0x2a
created by github.com/pg-sharding/spqr/router.(*InstanceImpl).RunAdm
        /Users/denchick/Code/spqr/router/instance.go:242 +0xdf

psql log:

➜  spqr git:(admin_console_imprv) psql "host=localhost port=7432"               

                SQPR router admin console
        Here you can configure your routing rules
------------------------------------------------
        You can find documentation here 
https://github.com/pg-sharding/spqr/tree/master/docs

psql (14.8 (Homebrew), server console)
Type "help" for help.

denchick=> lock key range krid1;
        lock key range        
------------------------------
 lock key range with id krid1
(1 row)

denchick=> lock key range krid1;
^CCancel request sent
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!?> 

Copy link
Collaborator

@Denchick Denchick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo

@reshke
Copy link
Collaborator Author

reshke commented Aug 4, 2023

todo

fixed

@Denchick
Copy link
Collaborator

Denchick commented Aug 7, 2023

Now the router really doesn't crash. unlock/lock really works. But when you try to lock for the second time, the client freezes, nothing happens in the router log either

{"{"level":"info","time":"2023-08-07T16:14:16+05:00","message":"console.Pro client start"}
{"level":"debug","client":824637407232,"message":{"Type":"Query","String":"lock key range krid1;"},"time":"2023-08-07T16:14:17+05:00","message":"client received message"}
{"level":"debug","query":"lock key range krid1;","type":"*spqrparser.Lock","time":"2023-08-07T16:14:17+05:00","message":"processQueryInternal: parsed query with type"}
{"{"level":"debug","key-range":"krid 1","time":"2023-08-07T16:14:17+05:00","message":"memqdb: lock key range"}

Is it better to explicitly throw an error or just allow the operation?

Copy link
Collaborator

@Denchick Denchick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed that it's great as it is.

@reshke reshke merged commit 0e44a1e into master Aug 7, 2023
8 checks passed
@reshke reshke deleted the admin_console_imprv branch August 16, 2023 08:43
CBists pushed a commit to CBists/spqr that referenced this pull request Nov 9, 2023
* Fix release

* Get tag in Makefile

Create coordinator.md (pg-sharding#225)

* Create coordinator.md

* update docs

* fix typo

---------

Co-authored-by: denchick <denchick@yandex-team.ru>

Run each feature in seperate job (pg-sharding#268)

Unite admin-console/psql-console pre-routing logic. (pg-sharding#259)

* Unite admin-console/psql-console pre-routing logic.

essentially fixes pg-sharding#255

* Fix tests

Update Router.md (pg-sharding#210)

* Update Router.md

resolves pg-sharding#201

* fix typo

---------

Co-authored-by: denchick <denchick@yandex-team.ru>

fix init.sql + backup.json logic (pg-sharding#262)

* fix init.sql + backup logic

* fix docs

* add new scenario

* fix spelling

Coordinator Show clients/pools/backend_connections funcs (pg-sharding#265)

* implement pools for coordinator
* `show pools` & `show backend connections ` now shows router names
* added units for coord pool
* added feature for show pools etc.

Show router status in `SHOW ROUTERS` command response (pg-sharding#266)

* Show router status in `SHOW ROUTERS` command responce

```

spqr-consolds=> show routers;
       show routers       | status
--------------------------+--------
 router r1-localhost:7010 | OPENED
(1 row)

```

* Linter fix

* Fix expected router out

* fixes

---------

Co-authored-by: denchick <denchick@yandex-team.ru>

Fix dataraces in memqdb (pg-sharding#260)

* fix datarace with memqdb.go?

* fix linter + ci

* add missing methods

* resolve conflicts

* fix deadlock: sharekey with unlockkey

* fix deadlock: dropkeyrange with unlockkeyrange

* fix Makefile

* refactor method

* fix comments

* fix error message

fix double lock key range on router when moving data (pg-sharding#271)

fix lock/unlock key range wrong namespace in qdb (pg-sharding#270)

* fix lock/unlock key range in coordinator

* fix e2e tests

test sync router after registration in coordinator (pg-sharding#267)

* test sync router after registration in coordinator

* check shard id in sync router test scenario

* fix sql result comparison in coordinator feature tests

Upload to existing release instead of creating a new one (pg-sharding#273)

add some checks to Split/Unite (pg-sharding#275)

* add some checks to UniteKeyRange in coordinator

* add some checks on bound to Split

* minor test scenarios titles fixes

Refactor QDB interfaces, add comments to code (pg-sharding#274)

check if sharding rule already exists, implement CheckKeyRangeLocked in etcdQDB (pg-sharding#278)

fix key range intersection checks (pg-sharding#276)

* fix moving key range to the same shardID

* fix adding key range with existing krid by splitting

* fix qdb key range intersection checks in qdb operations

* minor pr fixes

* fix tests

* extended tests

* cannot split range by lower or upper bound because range cannot be of 1 point

* fix comment

ping router before adding to qdb (pg-sharding#277)

* ping router before adding to qdb

* wrap ping router error

bump dependencies (pg-sharding#281)

fix coordinator crash when moving key range twice (pg-sharding#282)

Single coord (pg-sharding#280)

* locking coordinator

* made transactions atomic

* added tests

* fixed tests and linters

* fixed features

* enhansed coordinator lock

* fix

* requested changes

fix missing log level on router reload (pg-sharding#284)

Support X protocol in frontend interactions: Part 1 (pg-sharding#286)

* Support X protocol in frontend interactions

* Add mockgen and add test to Check TSA

* Add unit test to frontend function

* Fix linter

* Enhance makefile

* Bump lyx

* Comment frontend test as there is work to do, to make in work

Fix typo in spqr_test.go (pg-sharding#289)

Refactor Client, Relay and PoolMgr interfaces (pg-sharding#288)

* Refactor Client, Relay and PoolMgr interfaces

* Add test

* Add copy test

Parallelize scenarios (pg-sharding#285)

* Execute scenarios in feature in parallel
* not do second split

* cache images

from wal-g

* try without docker install

* don't build spqr-shard-image if docker/shard

doesn't changes

* reuse running jobs

* revert "cache images"

* refactor

add checks to move key range (pg-sharding#283)

* add checks to move key range

* change shardID from sh3 to non-existent in move feature test

Run regress tests on 12-15 Postgresql (pg-sharding#290)

* run regress on multiple version of postgres

* stop qdb after regress

* remove 16 and 17 version

Ignore feature tests artifacts (pg-sharding#293)

Added SCRAM auth feature tests (pg-sharding#292)

* Added "I run commandon host" step

* Added .feature file

* Some fixes

* Switch back to "regress" user

* Frontend auth test

* Added "Command output should match" step

* Added psql output check

* Added backend auth test

* Moved psql installation to Dockerfile

fix possible deadlock on big move (pg-sharding#291)

Bump lyx (pg-sharding#295)

* Bump lyx

* Regen

make gogen (pg-sharding#296)

Add gram rules for strart trace request (pg-sharding#297)

Run coordinator together with router (pg-sharding#287)

Dispatch coordinator goroutines with router binary start. Proxy all modifying requests to Coordinator via grpc.

Run admin console in proxy way for several type of queries:

    All modify requests to router console should be proxied to coordinator via particular gRPC-call. It is very dangerous to allow user to modify router metadata, like key-ranges or sharding rules, etc. This only should by done by coordinator or via coord. API.

    All show requests, like SHOW CLIENTS etc should be processed locally. In future, we can support installation-wide
    show requests processing using additional syntax, like SHOW CLIENTS is processed by router but SHOW GLOBAL CLIENTS to be proccesed by coordinator (coord. can process such queries already).

Regen yacc (pg-sharding#299)

Logproxy in router (pg-sharding#298)

Add workload recording infrastructure, workload recording API and bindings. Provide SQL interface to access new subsystem.

Replay test workload (pg-sharding#303)

Replay recorded workload in router.

revert config param (pg-sharding#300)

Update .gitignore (pg-sharding#304)

route begin transaction read-only to replica (pg-sharding#305)

* route begin transaction read-only to replica

* Bump lyx

* Bump

Bump lyx (pg-sharding#307)

bump lyx (pg-sharding#308)

Update main.go (pg-sharding#310)

Bump lyx (pg-sharding#311)

Xproto: extended query protocol interactions. (pg-sharding#302)

* Extended protocol interaction basic

* Fix

* Fix

* util+gomod

* Bump

* Fix

* Fix2

* Fix3

Bump lyx (pg-sharding#312)

* Bump luyx

* Fix

* Better

Added support for systemd notifications (pg-sharding#314)

* Added notifier type

* Added notifications to router instance

* Added alive notifications

* Added debug logging for systemd notifications

* Added reloading notifications support

* Added systemd notifier settings to config

* Fixed golangci-lint

Add client_encoding to router console (pg-sharding#315)

Add dataStyle to router console (pg-sharding#316)

Accept routing hints in comments (pg-sharding#317)

* Accept routing hints in comments

add dataspaces support

Change gram.y: Add DROP DATASPACE support to parser

Gen

Bugfix

Gen

delete DROP DATASPACE support

gen

Add show dataspace regress test

Small fix

fix regress tests and add tests to dataspace

Xproto: extended query protocol interactions. (pg-sharding#302)

* Extended protocol interaction basic

* Fix

* Fix

* util+gomod

* Bump

* Fix

* Fix2

* Fix3

Accept routing hints in comments (pg-sharding#317)

* Accept routing hints in comments

add dataspaces support

gen

hz

tests fix

add dataspaces support

Change gram.y: Add DROP DATASPACE support to parser

Gen

Bugfix

delete DROP DATASPACE support

fix regress tests and add tests to dataspace

Xproto: extended query protocol interactions. (pg-sharding#302)

* Extended protocol interaction basic

* Fix

* Fix

* util+gomod

* Bump

* Fix

* Fix2

* Fix3

Accept routing hints in comments (pg-sharding#317)

* Accept routing hints in comments

add dataspaces support

gen

tests fix

fix go.sum

gen

Delete .DS_Store

Delete .DS_Store files

add dataspaces support

Change gram.y: Add DROP DATASPACE support to parser

Gen

Bugfix

delete DROP DATASPACE support

Add show dataspace regress test

fix regress tests and add tests to dataspace

Xproto: extended query protocol interactions. (pg-sharding#302)

* Extended protocol interaction basic

* Fix

* Fix

* util+gomod

* Bump

* Fix

* Fix2

* Fix3

Accept routing hints in comments (pg-sharding#317)

* Accept routing hints in comments

add dataspaces support

gen

tests fix

Delete .DS_Store

gen
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Crash on lock cancel
2 participants