[build/dev] DSS local-instance cluster setup with haproxy #752
Conversation
build/dev/haproxy_local_setup.sh
Outdated
| DC_COMMAND=$* | ||
|
|
||
| if [[ "$DC_COMMAND" == "down" ]]; then | ||
| cleanup || true |
There was a problem hiding this comment.
Trying to call cleanup function created below to stop the instances, but this call is not working at the moment. Let me know what is the right way to call this function. call :cleanup is also not working.
There was a problem hiding this comment.
I think the function has to be declared before it is called in shell script
build/dev/haproxy_local_setup.sh
Outdated
| docker exec -it roacha cat haproxy.cfg > ~/Downloads/haproxy/haproxy.cfg | ||
|
|
||
| echo "Start the HAProxy container by mounting the cfg file." | ||
| docker run -d --name dss-crdb-cluster-for-testing --network dss_sandbox_default -p 26257:26257 -v ~/Downloads/haproxy:/usr/local/etc/haproxy:ro haproxy:1.7 |
There was a problem hiding this comment.
atm, I am using ~/Downloads/haproxy from local machine to mount the volume for haproxy. Let me know if I should be using something else. I tried using /var/tmp/, but it didn't work.
There was a problem hiding this comment.
Let's write haproxy.cfg to this folder (probably with a .gitignore entry to avoid committing to the repo) and then mount it as a single file: $(pwd)/haproxy.haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
build/dev/mutli_node_local_dss.md
Outdated
|
|
||
| # Testing | ||
|
|
||
| In a different window, run [`./check_dss.sh`](check_dss.sh) to run a |
There was a problem hiding this comment.
Invalid access token audience: localhost error is thrown while trying to run the test scripts. Is there anything I am missing while testing?
There was a problem hiding this comment.
On line 159 of haproxy_local_setup.sh, core-service is started with -accepted_jwt_audiences local-gateway which means it's expecting any access tokens it receives to have the audience local-gateway. The various check*.sh all generate tokens with localhost, I think (though some might use host.docker.internal; I haven't checked). Probably the easiest way to solve the issue is to also accept tokens intended for localhost with -accepted_jwt_audiences local-gateway,localhost
build/dev/haproxy_local_setup.sh
Outdated
| DC_COMMAND=$* | ||
|
|
||
| if [[ "$DC_COMMAND" == "down" ]]; then | ||
| cleanup || true |
There was a problem hiding this comment.
I think the function has to be declared before it is called in shell script
build/dev/haproxy_local_setup.sh
Outdated
| echo "Starting roacha with admin port on :8080" | ||
| docker run -d --rm --name roacha \ | ||
| -p 8080:8080 \ | ||
| "$FLAGS" > /dev/null |
There was a problem hiding this comment.
We have to remove the quotes here because otherwise we get the error unknown flag: --network dss_sandbox_default cockroachdb/cockroach:v21.2.3 start --insecure --join. To turn off the shell lint error, add # shellcheck disable=SC2086.
| docker rm -f dss-crdb-cluster-for-testing &> /dev/null || true | ||
|
|
||
| echo "Starting roacha with admin port on :8080" | ||
| docker run -d --rm --name roacha \ |
There was a problem hiding this comment.
I'm getting an error here because network dss_sandbox_default not found -- if we want to create the network first, we need docker network create dss_sandbox_default (this network is created automatically when the docker-compose run_locally system is brought up, but it is deleted when that system is brought down)
build/dev/haproxy_local_setup.sh
Outdated
| docker exec -it roacha cat haproxy.cfg > ~/Downloads/haproxy/haproxy.cfg | ||
|
|
||
| echo "Start the HAProxy container by mounting the cfg file." | ||
| docker run -d --name dss-crdb-cluster-for-testing --network dss_sandbox_default -p 26257:26257 -v ~/Downloads/haproxy:/usr/local/etc/haproxy:ro haproxy:1.7 |
There was a problem hiding this comment.
Let's write haproxy.cfg to this folder (probably with a .gitignore entry to avoid committing to the repo) and then mount it as a single file: $(pwd)/haproxy.haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
build/dev/mutli_node_local_dss.md
Outdated
|
|
||
| # Testing | ||
|
|
||
| In a different window, run [`./check_dss.sh`](check_dss.sh) to run a |
There was a problem hiding this comment.
On line 159 of haproxy_local_setup.sh, core-service is started with -accepted_jwt_audiences local-gateway which means it's expecting any access tokens it receives to have the audience local-gateway. The various check*.sh all generate tokens with localhost, I think (though some might use host.docker.internal; I haven't checked). Probably the easiest way to solve the issue is to also accept tokens intended for localhost with -accepted_jwt_audiences local-gateway,localhost
|
|
||
| Setting up HAProxy requires generating a configuration file by running `cockroach gen haproxy` on one of the cluster nodes. that is preset to work with the running cluster. Generated `haproxy.cfg` file is then mounted to HAProxy container via local machine's ~/Download/haproxy/ folder. | ||
|
|
||
| # Testing |
There was a problem hiding this comment.
The important thing about this setup is that it enables resiliency to loss of database nodes. Let's add documentation to minimally demonstrate that. I would expect additional steps to kill a node, show the system still works, restore that node and kill a different one, show the system still works, etc
|
Also, let's add a note in the documentation that the system can be cleaned up with |
BenjaminPelletier
changed the title
No description provided.