Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update the c2c example script to point to Replicator #874

Merged
merged 2 commits into from
May 24, 2024
Merged

Update the c2c example script to point to Replicator #874

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cf50d83
Update the c2c example script to point to Replicator
BramGruneir May 14, 2024
2e72fdf
Merge branch 'master' into bram_renames4
sravotto May 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 53 additions & 41 deletions scripts/docker_c2c/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,27 @@
# Docker c2c demo w/ Monitoring

Simple script to compose docker environment with:
+ running MovR application
+ source cluster :: http://localhost:8080
+ target cluster :: http://localhost:8082
+ cdc-sink :: `--bindAddr :30004 --metricsAddr :30005`
+ prometheus :: http://localhost:9090
+ grafana :: http://localhost:3000/dashboards
+ `admin/cdc-sink`

* running MovR application
* source cluster :: <http://localhost:8080>
* target cluster :: <http://localhost:8082>
* replicator :: `--bindAddr :30004 --metricsAddr :30005`
* prometheus :: <http://localhost:9090>
* grafana :: <http://localhost:3000/dashboards>
* `admin/replicator`

## Installation
The demo scripts exist in the github repository for `cdc-sink`.
+ `git clone git@github.com:cockroachdb/cdc-sink.git`

The demo scripts exist in the github repository for Replicator.

* `git clone git@github.com:cockroachdb/replicator.git`
...or...
+ `gh repo clone cockroachdb/cdc-sink`
* `gh repo clone cockroachdb/replicator`

```bash
$ git clone git@github.com:cockroachdb/cdc-sink.git
$ git clone git@github.com:cockroachdb/replicator.git

Cloning into 'cdc-sink'...
Cloning into 'replicator'...
remote: Enumerating objects: 5198, done.
remote: Counting objects: 100% (714/714), done.
remote: Compressing objects: 100% (365/365), done.
Expand All @@ -28,46 +31,55 @@ Resolving deltas: 100% (3556/3556), done.
```

## Run the Demo

Move into the `docker_c2c` directory:

```bash
$ cd cdc-sink/scripts/docker_c2c
cd replicator/scripts/docker_c2c
```
The underling `grafana`, `prometheus` and `scripts` directories are all needed to run the `run_compose.sh` script.

The underling `grafana`, `prometheus` and `scripts` directories are all needed to run the
`run_compose.sh` script.

```bash
glenn ~/git/cdc-sink/scripts/docker_c2c [new_scripts] $ ls
README.md docker-compose.yml grafana prometheus run_compose.sh scripts
glenn ~/git/replicator/scripts/docker_c2c [new_scripts] $ ls
README.md docker-compose.yml grafana prometheus run_compose.sh scripts
```

You will need to set the `COCKROACH_DEV_LICENSE` and `COCKROACH_DEV_ORGANIZATION` environment variables.
You will need to set the `COCKROACH_DEV_LICENSE` and `COCKROACH_DEV_ORGANIZATION` environment
variables.

```bash
export COCKROACH_DEV_LICENSE="_your_license_key"
export COCKROACH_DEV_ORGANIZATION="_your_organization_name"
```
The environment variables are used to create a `crdb_env` which will be used by the [setup.sh](scripts/setup.sh) script when initializing the clusters and workloads.

Finally, run the following to compose c2c Docker Demo::
+ `./run_compose.sh`
The environment variables are used to create a `crdb_env` which will be used by the
[setup.sh](scripts/setup.sh) script when initializing the clusters and workloads.

Finally, run the following to compose c2c Docker Demo:

* `./run_compose.sh`

```bash
$ ./run_compose.sh
[+] Running 21/21
✔ Network docker_c2c_default Created 0.2s
✔ Volume "docker_c2c_roach_source" Created 0.0s
✔ Volume "docker_c2c_roach_target" Created 0.0s
✔ Volume "docker_c2c_backup" Created 0.0s 0.0s
✔ Container docker_c2c-roach_target-1 Started 0.0s
✔ Container docker_c2c-roach_source-1 Started 0.0s
✔ Container docker_c2c-prometheus-1 Started 0.0s
✔ Container docker_c2c-roach_target_initsleep-1 Exited 0.0s
✔ Container docker_c2c-grafana-1 Started 0.0s
✔ Container docker_c2c-roach_scripts-1 Exited 0.0s
✔ Container docker_c2c-roach_source_create_feed-1 Exited 0.0s
✔ Container docker_c2c-cdc-sink-1 Started 0.0s
✔ Container docker_c2c-roach_source_movr_run-1 Started 0.0s
✔ Network docker_c2c_default Created 0.2s
✔ Volume "docker_c2c_roach_source" Created 0.0s
✔ Volume "docker_c2c_roach_target" Created 0.0s
✔ Volume "docker_c2c_backup" Created 0.0s
✔ Container docker_c2c-roach_target-1 Started 0.0s
✔ Container docker_c2c-roach_source-1 Started 0.0s
✔ Container docker_c2c-prometheus-1 Started 0.0s
✔ Container docker_c2c-roach_target_initsleep-1 Exited 0.0s
✔ Container docker_c2c-grafana-1 Started 0.0s
✔ Container docker_c2c-roach_scripts-1 Exited 0.0s
✔ Container docker_c2c-roach_source_create_feed-1 Exited 0.0s
✔ Container docker_c2c-replicator-1 Started 0.0s
✔ Container docker_c2c-roach_source_movr_run-1 Started 0.0s
Resetting Grafana Admin Password....

docker exec -it 2f92d862ae60 /usr/share/grafana/bin/grafana cli --homepath /usr/share/grafana admin reset-admin-password cdc-sink
docker exec -it 2f92d862ae60 /usr/share/grafana/bin/grafana cli --homepath /usr/share/grafana admin reset-admin-password replicator
INFO [10-19|23:05:02] Starting Grafana logger=settings version= commit= branch= compiled=1970-01-01T00:00:00Z
INFO [10-19|23:05:02] Config loaded from logger=settings file=/usr/share/grafana/conf/defaults.ini
INFO [10-19|23:05:02] Config overridden from Environment variable logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
Expand All @@ -88,15 +100,15 @@ INFO [10-19|23:05:02] Envelope encryption state logger=secrets en

Admin password changed successfully ✔

Grafana login is: admin/cdc-sink
Grafana login is: admin/replicator
Grafana URL: http://localhost:3000/login
```

Once this has completed, the demo will be running in detached mode and all the components can be accessed:
+ source cluster :: http://localhost:8080
+ target cluster :: http://localhost:8082
+ cdc-sink :: `--bindAddr :30004 --metricsAddr :30005`
+ prometheus :: http://localhost:9090
+ grafana :: http://localhost:3000/dashboards
+ `admin/cdc-sink`

* source cluster :: <http://localhost:8080>
* target cluster :: <http://localhost:8082>
* replicator :: `--bindAddr :30004 --metricsAddr :30005`
* prometheus :: <http://localhost:9090>
* grafana :: <http://localhost:3000/dashboards>
* `admin/replicator`
12 changes: 6 additions & 6 deletions scripts/docker_c2c/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ services:
hostname: roach_source
image: cockroachdb/cockroach:latest-v23.1
ports:
- "26257:26257"
- "26257:26257"
- "8080:8080"
command: start-single-node --insecure --external-io-dir /backup
volumes:
Expand Down Expand Up @@ -66,15 +66,15 @@ services:
roach_target_initsleep:
condition: service_completed_successfully

cdc-sink:
hostname: cdc-sink
image: cockroachdb/cdc-sink:master
replicator:
hostname: replicator
image: cockroachdb/replicator:master
ports:
- "30005:30005"
depends_on:
roach_scripts:
condition: service_completed_successfully
command: start --bindAddr :30004 --metricsAddr :30005 --tlsSelfSigned --disableAuthentication --targetConn 'postgresql://root@roach_target:26257/?sslmode=disable' --selectBatchSize 100 --foreignKeys
command: start --bindAddr :30004 --metricsAddr :30005 --tlsSelfSigned --disableAuthentication --targetConn 'postgresql://root@roach_target:26257/?sslmode=disable' --selectBatchSize 100 --foreignKeys
restart: always

roach_source_create_feed:
Expand All @@ -83,7 +83,7 @@ services:
depends_on:
roach_scripts:
condition: service_completed_successfully
command: sql --url 'postgresql://root@roach_source:26257/?sslmode=disable' --insecure --execute "CREATE CHANGEFEED FOR TABLE movr.users,movr.vehicles,movr.rides,movr.vehicle_location_histories,movr.promo_codes,movr.user_promo_codes INTO 'webhook-https://cdc-sink:30004/movr/public?insecure_tls_skip_verify=true' WITH updated, resolved='10s';"
command: sql --url 'postgresql://root@roach_source:26257/?sslmode=disable' --insecure --execute "CREATE CHANGEFEED FOR TABLE movr.users,movr.vehicles,movr.rides,movr.vehicle_location_histories,movr.promo_codes,movr.user_promo_codes INTO 'webhook-https://replicator:30004/movr/public?insecure_tls_skip_verify=true' WITH updated, resolved='10s';"
restart: on-failure

roach_source_movr_run:
Expand Down
10 changes: 5 additions & 5 deletions scripts/docker_c2c/run_compose.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,24 +21,24 @@

if [ -z "COCKROACH_DEV_LICENSE" ] || [ -z "COCKROACH_DEV_ORGANIZATION" ]
then
echo "The COCKROACH_DEV_LICENSE and COCKROACH_DEV_ORGANIZATION env variables must be set to run CRDB Changefeeds needed for cdc-sink"
echo "The COCKROACH_DEV_LICENSE and COCKROACH_DEV_ORGANIZATION env variables must be set to run CRDB Changefeeds needed for replicator"
exit
fi

echo "export COCKROACH_DEV_LICENSE='"$COCKROACH_DEV_LICENSE"'" > scripts/crdb_env
echo "export COCKROACH_DEV_ORGANIZATION='"$COCKROACH_DEV_ORGANIZATION"'" >> scripts/crdb_env

docker-compose up --detach
docker-compose up --detach

echo "Resetting Grafana Admin Password...."
echo ""
export cid=`docker ps |grep grafana- |awk '{printf("%s",$1)}'`

echo "docker exec -it ${cid} /usr/share/grafana/bin/grafana cli --homepath /usr/share/grafana admin reset-admin-password cdc-sink"
docker exec -it ${cid} /usr/share/grafana/bin/grafana cli --homepath "/usr/share/grafana" admin reset-admin-password cdc-sink
echo "docker exec -it ${cid} /usr/share/grafana/bin/grafana cli --homepath /usr/share/grafana admin reset-admin-password replicator"
docker exec -it ${cid} /usr/share/grafana/bin/grafana cli --homepath "/usr/share/grafana" admin reset-admin-password replicator
RESULT=$?
if [ $RESULT -eq 0 ]; then
echo "Grafana login is: admin/cdc-sink"
echo "Grafana login is: admin/replicator"
else
echo "Grafana password was not reset. Must do it manually... (admin/admin) is the initial password"
fi
Expand Down