From 19b6bc9c152b2502e3ab70056e0047ab00bb7425 Mon Sep 17 00:00:00 2001 From: Nick Zelei <2420177+nickzelei@users.noreply.github.com> Date: Wed, 3 Jan 2024 00:29:30 +0430 Subject: [PATCH] Nick/neos 582 create compose file that uses production images (#1015) --- README.md | 164 +++++++++++------- compose/README.md | 10 ++ compose/compose-auth-prod.yml | 145 ++++++++++++++++ compose/compose-auth.yml | 1 + compose/compose-prod.yml | 92 ++++++++++ docs/docs/deploy/docker-compose.md | 22 ++- .../web/app/api/auth/[...nextauth]/auth.ts | 1 - 7 files changed, 359 insertions(+), 76 deletions(-) create mode 100644 compose/compose-auth-prod.yml create mode 100644 compose/compose-prod.yml diff --git a/README.md b/README.md index e8027217f..b4cb860b1 100644 --- a/README.md +++ b/README.md @@ -59,37 +59,73 @@ Our mission is to help developers build better, more resilient applications whil ## Getting started -You can also check out our [Docs](https://docs.neosync.dev) for more guides including a production-ready guide. Note: these are still a work in progress. +You can also check out our [Docs](https://docs.neosync.dev) for more guides including a production-ready guide. ## Run Neosync locally -To set up and run Neosync locally, make sure you have Git and Docker installed on your system. +Neosync is a fully dockerized setup. Due to this, there are many different ways to run Neosync. -The sections below detail the tools required to build and run the Neosync development environment. -This can be circumvented by using our official devcontainer which comes pre-installed with all of the tools necessary. +There are three officially supported ways of running Neosync locally: -There are then two ways to start Neosync: +1. Bare Metal +2. `docker compose` +3. Kubernetes via Tilt and Kind. -- Tilt -- Docker Compose +For more in-depth details on environment variables as well as Kubernetes deployments, check out the [Deploy Neosync](https://docs.neosync.dev/deploy/introduction) section of our Docs. -Tilt is the method we currently use to development Neosync. This lets us develop as if we are running inside of a Kubernetes cluster. -This isn't for everyone, which is we also offer a compose method for a simpler, kubernetesless approach. +This readme will focus more on the development environment and simple steps to getting Neosync up on your system. -Check out the sections below for which method applies to you. +### Simply trying Neosync + +If you just want to try out Neosync to see what it's like or get a feel for the product, most of the development setup guide below can be skipped. +We provide a `compose.yml` file that contains production image references that allow you to get up and running with just a few commands without having to build anything on your system. + +The simplest configuration of Neosync is standing it up without any form of authentication. +This can be done with the following commands: + +```sh +$ docker compose -f temporal/compose.yml up -d +$ docker compose -f compose/compose-prod.yml up -d +``` + +Neosync will now be available on [http://localhost:3000](http://localhost:3000). + +A compose file is also provided that stands up [Keycloak](https://keycloak.org), an open source auth solution. + +To stand up Neosync with auth, simply run the following commands: + +```sh +$ docker compose -f temporal/compose.yml up -d +$ docker compose -f compose/compose-auth-prod.yml up -d +``` + +Neosync will now be available on [http://localhost:3000](http://localhost:3000) with authentication pre-configured! +Click the login with Keycloak button, register an account (locally) and you'll be logged in! + +### Neosync Development Environment + +This section goes into detail each tool that is used for development Neosync. +This section casts a wide net, and some tools may not be required depending on if you are using a Tilt setup or a Compose setup. +Most of the `Kubernetes` focused tools can be skipped if develoing via compose. + +### Neosync DevContainer + +Neosync has a pre-published [devcontainer](https://containers.dev/) that can be used to easily get a working Neosync dev environment. +This container comes pre-packaged with all of the tools needed for developing Neosync, and works with Tilt or Compose, or Bare Metal setups. ### Tools -Currently, the primary development environment is done by deploying the app and its dependent resources into a `kind` cluster using `tilt`. -We utilize `helm` charts to wrap up deployable artifacts and have `tilt` install these to closely mimic a production environment. -Due to this, there are a number of dependencies that must be installed on a host system prior to being able to run `neosync` locally. +This section contains a flat list of the tools that are used to develop Neosync and why. Detailed below are the main dependencies are descriptions of how they are utilized: #### Kubernetes -Kubernetes is used today as our primary development environment. Tilt is a great tool that lets you define your environment in code. -This lets us develop quickly, locally, while closely mimicking a real production environment. +If you're choosing to develop in a Tilt environment, this section is more important as it contains all of the K8s focused tooling. + +Tilt is a great tool that is used to automate the setup of a Kubernetes cluster. There are multiple `Tiltfile`'s througout the code, along with a top-level one that is used to inject all of the K8s manifests to setup Neosync inside of a K8s cluster. + +This enables fast development, locally, while closely mimicking a real production environment. - [kind](https://github.com/kubernetes-sigs/kind) - Kubernetes in Docker. We use this to spin up a barebones kubernetes cluster that deploys all of the `neosync` resources. @@ -106,9 +142,9 @@ This lets us develop quickly, locally, while closely mimicking a real production - [helmfile](https://github.com/helmfile/helmfile) - Declaratively define a helmfile in code! We have all of our dev charts defined as a helmfile, of which Tilt points directly to. -#### Golang + Protobuf +#### Go + Protobuf -- Golang +- [Go](https://go.dev/) - The language of choice for our backend and worker packages - [sqlc](https://github.com/sqlc-dev/sqlc) - Our tool of choice for the data-layer. This lets us write pure SQL and let sqlc generate the rest. @@ -116,17 +152,18 @@ This lets us develop quickly, locally, while closely mimicking a real production - Our tool of choice for interfacing with protobuf - [golangci-ci](https://github.com/golangci/golangci-lint) - The golang linter of choice +- [migrate](https://github.com/golang-migrate/migrate) + - Golang Migrate is the tool that is used to run DB Migrations for the API. #### Npm/Nodejs -- Node/Npm +- [Node/Npm](https://nodejs.org/en) + - Used to run the app, along with Nextjs. All of these tools can be easily installed with `brew` if on a Mac. Today, `sqlc` and `buf` don't need to be installed locally as we exec docker images for running them. This lets us declare the versions in code and docker takes care of the rest. -It's of course possible run everything on bare metal without Kuberentes or Tilt, but there will be more work getting everything up and running (at least today). - ### Brew Install Each tool above can be straightforwardly installed with brew if on Linux/MacOS @@ -135,81 +172,74 @@ Each tool above can be straightforwardly installed with brew if on Linux/MacOS brew install kind tilt-dev/tap/tilt tilt-dev/tap/ctlptl kubernetes-cli kustomize helm helmfile go sqlc buf golangci-lint node ``` -### Devcontainer - -Host machine setup can be skipped by developing inside of a vscode devcontainer. -This container comes pre-baked with all of the tools we use to develop and work on neosync. -This container also supports running neosync with compose or tilt. - -### Running Docker with Docker Desktop - -When running with either `Tilt` or `docker compose`, we map volumes from these filesystems to the host machine for both neosync and Temporal's databases. -We mount a container path locally in a `.data` folder. If on a Mac, ensure that you've allowed wherever this repository has been cloned into to the allow-list in Docker Desktop. - -The allow list can be found by first opening Docker Desktop. `Settings -> Resources -> File Sharing` and add the path to the Neosync repository. - -If you don't want to do this, you can remove the volume mappings in the compose file or remove the pvc for Tilt. -This comes at a negative of the local database not surviving restarts, however. +### Setup with Compose -### Setup with Tilt +When running with either `Tilt` or `docker compose`, volumes are mapped from these filesystems to the host machine for both neosync and Temporal's databases. +A volume is mounted locally in a `.data` folder. -Step 1 is to ensure that the `kind` cluster is up and running along with its registry. -This can be manually created, or done simply with `ctlptl`. -The cluster is declaratively defined [here](./tilt/kind/cluster.yaml) +To enable hot reloading, must run `docker compose watch` instead of `up`. **Currently there is a limitation with devcontainers where this command must be run via `sudo`.** +This works pretty well with the `app`, but can be a bit buggy with the `api` or `worker`. +Sometimes it's a little easier to just rebuild the docker container like. -The below command invokes the cluster-create script that can be found [here](./tilt/scripts/cluster-create.sh) +Assuming the latest binary is available in the bin folder: ``` -make cluster-create +$ docker compose up -d --build api ``` -After the cluster has been successfully created, `tilt up` can be run to start up `neosync`. -Refer to the top-level [Tiltfile](./Tiltfile) for a clear picture of everything that runs. -Each dependency in the `neosync` repo is split into sub-Tiltfiles so that they can be run in isolation, or in combination with other sub-resources more easily. - -Once everything is up and running, the app can be accessed at locally at `http://localhost:3000`. - -### Setup with Docker Compose - -Neosync can be run with compose. This works pretty well, but is a bit more manual today than with Tilt. -Not everything is hot-reload, but you can successfully run everything using just compose instead of having to manage a kubernetes cluster and running Tilt. -To enable hot reloading, must run `docker compose watch` instead of `up`. **Currently there is a limitation with devcontainers where this command must be run via `sudo`.** - -There are two compose files that need to be run today. The first is the Temporal compose, the second is the Neosync compose. -It's suggested you run these separate (as of today) for a clean separation of concerns. - #### Building the backend and worker when using Docker Compose. -Prior to running `docker compose up -d`, the worker and api will need to be built. +If using the dev-focused compose instead of the `*-prod.yml` compose files, the binaries for the `api` and `worker` will need to be built. When building the Go processes with the intention to run with `docker compose`, it's important to run `make dbuild` instead of the typical `make build` so that the correct `GOOS` is specified. This is only needed if your native OS is not Linux (or aren't running in a devcontainer). The `make dbuild` command ensures that the Go binary is compiled for Linux instead of the host os. This will need to be done for both the `worker` and `api` processes prior to running compose up. -#### Running Compose - -``` +```sh $ docker compose -f temporal/compose.yml up -d $ docker compose -f compose.yml up -d ``` -Once everything is up and running, the app can be accessed locally at `http://localhost:3000`. - -Work to be done: - -- inherit the temporal compose inside of the neosync compose, separate with compose profiles. +Once everything is up and running, the app can be accessed locally at [http://localhost:3000](http://localhost:3000). #### Running Compose with Authentication Note, a compose file with authentication pre-configured can be found [here](./compose/compose-auth.yml). -This will stand up Keycloak with a pre-configured realm that will allow logging in to Neosync with a standard username and password, competely offline! +This will stand up Keycloak with a pre-configured realm that will allow logging in to Neosync with a standard username and password, completely offline! -``` +```sh $ docker compose -f temporal/compose.yml up -d $ docker compose -f compose/compose-auth.yml up -d ``` +#### Docker Desktop + +If using Docker Desktop, the host file path to the `.data` folder will need to be added to the File Sharing tab. + +The allow list can be found by first opening Docker Desktop. `Settings -> Resources -> File Sharing` and add the path to the Neosync repository. + +If you don't want to do this, the volume mappings can be removed from the compose file, or by removing the PVC for Tilt. +This comes at a negative of the local database not surviving restarts. + +### Setup with Tilt + +Step 1 is to ensure that the `kind` cluster is up and running along with its registry. +This can be manually created, or done simply with `ctlptl`. +The cluster is declaratively defined [here](./tilt/kind/cluster.yaml) + +The below command invokes the cluster-create script that can be found [here](./tilt/scripts/cluster-create.sh) + +``` +make cluster-create +``` + +After the cluster has been successfully created, `tilt up` can be run to start up `neosync`. +Refer to the top-level [Tiltfile](./Tiltfile) for a clear picture of everything that runs. +Each dependency in the `neosync` repo is split into sub-Tiltfiles so that they can be run in isolation, or in combination with other sub-resources more easily. + +Once everything is up and running, the app can be accessed locally at [http://localhost:3000](http://localhost:3000). + ## Resources Some resources to help you along the way: diff --git a/compose/README.md b/compose/README.md index 3d08f2803..fb10ed44d 100644 --- a/compose/README.md +++ b/compose/README.md @@ -6,6 +6,16 @@ Alternative Compose files that can be used to stand up additional services, or a Everything in the root `compose.yml` but it also enables authentication through the use of Keycloak. +## compose-auth-prod.yml + +This is the same as `compose-auth.yml` but it uses pre-packaged Docker images. +This is a good compose file to run if you just want to try out Neosync (with auth) and not worry about a build environment. All that is needed is Docker! + +## compose-prod.yml + +This is the same as `compose.yml` (or `compose-auth.yml` minus auth) but it uses pre-packaged Docker images. +This is a good compose file to run if you just want to try out Neosync (without auth) and not worry about a build environment. All that is needed is Docker! + ## compose-db.yml Stands up two separate postgres databases that can be used for testing Neosync. diff --git a/compose/compose-auth-prod.yml b/compose/compose-auth-prod.yml new file mode 100644 index 000000000..c30f61b80 --- /dev/null +++ b/compose/compose-auth-prod.yml @@ -0,0 +1,145 @@ +services: + app: + container_name: neosync-app + image: ghcr.io/nucleuscloud/neosync/app + ports: + - 3000:3000 + environment: + - NUCLEUS_ENV=dev + - NEXTAUTH_SECRET=foo + - NEXTAUTH_URL=http://localhost:3000 + - NEOSYNC_API_BASE_URL=http://api:8080 + - NEXT_PUBLIC_APP_BASE_URL=http://localhost:3000 + + - POSTHOG_KEY=phc_qju45RhNvCDwYVdRyUjtWuWsOmLFaQZi3fmztMBaJip + + - AUTH_ENABLED=true + - AUTH_ISSUER=http://keycloak:8080/realms/neosync + - AUTH_EXPECTED_ISSUER=http://localhost:8083/realms/neosync + - AUTH_AUTHORIZE_URL=http://localhost:8083/realms/neosync/protocol/openid-connect/auth + - AUTH_USERINFO_URL=http://keycloak:8080/realms/neosync/protocol/openid-connect/userinfo + - AUTH_TOKEN_URL=http://keycloak:8080/realms/neosync/protocol/openid-connect/token + + - AUTH_CLIENT_ID=neosync-app + - AUTH_CLIENT_SECRET=72alWGzhHInDskRHduTQ8BjB4Lgn0n3a + - AUTH_AUDIENCE=neosync + - AUTH_SCOPE=openid email profile offline_access + - AUTH_PROVIDER_ID=keycloak + - AUTH_PROVIDER_NAME=Keycloak + + networks: + - neosync-network + - kc-network + + db: + container_name: neosync-db + image: postgres:15 + ports: + - 5432:5432 + environment: + - POSTGRES_DB=nucleus + - POSTGRES_PASSWORD=foofar + - PGUSER=postgres + healthcheck: + test: ["CMD", "pg_isready"] + interval: 10s + timeout: 5s + retries: 5 + networks: + - neosync-network + - kc-network + volumes: + - $PWD/.data/neosync-postgres/data:/var/lib/postgresql/data + - $PWD/hack/pg-init:/docker-entrypoint-initdb.d + + api: + container_name: neosync-api + image: ghcr.io/nucleuscloud/neosync/api + ports: + - 8080:8080 + command: serve connect + environment: + - HOST=0.0.0.0 + - PORT=8080 + - NUCLEUS_ENV=dev + - TEMPORAL_URL=temporal:7233 + - DB_AUTO_MIGRATE=true + - DB_SCHEMA_DIR=/migrations + - DB_HOST=db + - DB_PORT=5432 + - DB_NAME=nucleus + - DB_USER=postgres + - DB_PASS=foofar + - DB_SSL_DISABLE=true + - DB_MIGRATIONS_TABLE=neosync_api_schema_migrations + - DB_MIGRATIONS_TABLE_QUOTED=false + + - AUTH_ENABLED=true + - AUTH_AUDIENCE=neosync + - AUTH_BASEURL=http://keycloak:8080/realms/neosync + - AUTH_EXPECTED_ISS=http://localhost:8083/realms/neosync + - AUTH_CLI_CLIENT_ID=neosync-cli + - AUTH_CLI_AUDIENCE=neosync + - AUTH_CLIENTID_SECRET={"neosync-cli":"GkVsthDzDvBfzb2vT4UO95xbXrwoXE5w"} + - AUTH_SIGNATURE_ALGORITHM=RS256 + + networks: + - neosync-network + - temporal-network + - kc-network + + depends_on: + db: + condition: service_healthy + restart: true + + worker: + container_name: neosync-worker + image: ghcr.io/nucleuscloud/neosync/worker + environment: + - NUCLEUS_ENV=dev + - TEMPORAL_URL=temporal:7233 + - TEMPORAL_NAMESPACE=default + - TEMPORAL_TASK_QUEUE=sync-job + - NEOSYNC_URL=http://api:8080 + networks: + - neosync-network + - temporal-network + + keycloak: + container_name: neosync-auth-keycloak + image: quay.io/keycloak/keycloak + ports: + - 8083:8080 + command: start-dev --import-realm + environment: + - KEYCLOAK_ADMIN=admin + - KEYCLOAK_ADMIN_PASSWORD=change_me + + - KC_HOSTNAME_URL=http://localhost:8083 + + - KC_DB=postgres + - KC_DB_URL_HOST=db + - KC_DB_URL_PORT=5432 + - KC_DB_URL_DATABASE=nucleus + - KC_DB_USERNAME=postgres + - KC_DB_PASSWORD=foofar + - KC_DB_SCHEMA=keycloak + volumes: + - $PWD/keycloak/imports:/opt/keycloak/data/import + networks: + - kc-network + depends_on: + db: + condition: service_healthy + restart: true + +networks: + neosync-network: + driver: bridge + name: neosync-network + kc-network: + driver: bridge + name: kc-network + temporal-network: + external: true diff --git a/compose/compose-auth.yml b/compose/compose-auth.yml index a4b310816..586d80473 100644 --- a/compose/compose-auth.yml +++ b/compose/compose-auth.yml @@ -10,6 +10,7 @@ services: environment: - NUCLEUS_ENV=dev - NEXTAUTH_SECRET=foo + - NEXTAUTH_URL=http://localhost:3000 - NEOSYNC_API_BASE_URL=http://api:8080 - NEXT_PUBLIC_APP_BASE_URL=http://localhost:3000 diff --git a/compose/compose-prod.yml b/compose/compose-prod.yml new file mode 100644 index 000000000..f7e9a45ad --- /dev/null +++ b/compose/compose-prod.yml @@ -0,0 +1,92 @@ +services: + app: + container_name: neosync-app + image: ghcr.io/nucleuscloud/neosync/app + ports: + - 3000:3000 + environment: + - NUCLEUS_ENV=dev + - NEXTAUTH_SECRET=foo + - NEXTAUTH_URL=http://localhost:3000 + - NEOSYNC_API_BASE_URL=http://api:8080 + - NEXT_PUBLIC_APP_BASE_URL=http://localhost:3000 + + - POSTHOG_KEY=phc_qju45RhNvCDwYVdRyUjtWuWsOmLFaQZi3fmztMBaJip + + - AUTH_ENABLED=false + + networks: + - neosync-network + + db: + container_name: neosync-db + image: postgres:15 + ports: + - 5432:5432 + environment: + - POSTGRES_DB=nucleus + - POSTGRES_PASSWORD=foofar + - PGUSER=postgres + healthcheck: + test: ["CMD", "pg_isready"] + interval: 10s + timeout: 5s + retries: 5 + networks: + - neosync-network + volumes: + - $PWD/.data/neosync-postgres/data:/var/lib/postgresql/data + - $PWD/hack/pg-init:/docker-entrypoint-initdb.d + + api: + container_name: neosync-api + image: ghcr.io/nucleuscloud/neosync/api + ports: + - 8080:8080 + command: serve connect + environment: + - HOST=0.0.0.0 + - PORT=8080 + - NUCLEUS_ENV=dev + - TEMPORAL_URL=temporal:7233 + - DB_AUTO_MIGRATE=true + - DB_SCHEMA_DIR=/migrations + - DB_HOST=db + - DB_PORT=5432 + - DB_NAME=nucleus + - DB_USER=postgres + - DB_PASS=foofar + - DB_SSL_DISABLE=true + - DB_MIGRATIONS_TABLE=neosync_api_schema_migrations + - DB_MIGRATIONS_TABLE_QUOTED=false + + - AUTH_ENABLED=false + + networks: + - neosync-network + - temporal-network + + depends_on: + db: + condition: service_healthy + restart: true + + worker: + container_name: neosync-worker + image: ghcr.io/nucleuscloud/neosync/worker + environment: + - NUCLEUS_ENV=dev + - TEMPORAL_URL=temporal:7233 + - TEMPORAL_NAMESPACE=default + - TEMPORAL_TASK_QUEUE=sync-job + - NEOSYNC_URL=http://api:8080 + networks: + - neosync-network + - temporal-network + +networks: + neosync-network: + driver: bridge + name: neosync-network + temporal-network: + external: true diff --git a/docs/docs/deploy/docker-compose.md b/docs/docs/deploy/docker-compose.md index 83f862083..5c5f2579f 100644 --- a/docs/docs/deploy/docker-compose.md +++ b/docs/docs/deploy/docker-compose.md @@ -7,20 +7,22 @@ slug: /deploy/docker-compose ## Deploying with Docker Compose -A `compose.yml` file is provided in the root of the repository to easily and quickly get Neosync up and running. -This compose file is currently tailored for localhost/development environments, but should be easily adaptable to production environments. +A `compose.yml` file is provided in the `compose` folder to easily and quickly get Neosync up and running without having to build any dependencies. All that's needed is `docker`. -There is a companion compose file found in the `temporal` folder that should be run prior to the main `compose.yml` to stand up a development instance of Temporal and all of it's dependencies. +This compose file is not tailed for development environments as it uses pre-baked docker images that are intended for use in production environments. + +There is a companion compose file found in the `temporal` folder that should be run prior to the main `compose.yml` to stand up an instance of Temporal and all of it's dependencies. + +To simplify the setup even further, a compose file is provided that does not stand up authentication for Neosync. Check out the section below to stand up Neosync with auth. -Due to being a development focused setup, both Temporal and Neosync are started up without Authentication enabled. This is done to simplify configuration and standup time. Database volumes for both Tempral and Neosync are mapped to the `.data` folder inside of the repository. Note: If using Docker Desktop, this folder will have to be added to the list of allowed file system mappings prior to running docker compose. ```sh $ docker compose -f temporal/compose.yml up -d -$ docker compose up -d +$ docker compose -f compose/compose-prod.yml up -d ``` -Once all of the containers come online, the app is now routable via `http://localhost:3000`. +Once all of the containers come online, the app is now routable via [http://localhost:3000](http://localhost:3000). ## Deploy with Docker Compose and Authentication @@ -28,12 +30,16 @@ Neosync provides an auth friendly compose file that will stand up Neosync in aut ```sh $ docker compose -f temporal/compose.yml up -d -$ docker compose -f compose/compose-auth.yml up -d +$ docker compose -f compose/compose-auth-prod.yml up -d ``` -It comes default with two clients that allow the app and cli to login successfully. +Keycloak comes default with two clients that allow the app and cli to login successfully. On first boot up, Keycloak will assert itself with the provided realm Neosync realm. When navigating to Neosync for the first time, you'll land on the Keycloak signin page. It is easy to create an account simply by going through the register flow. This will persist restarts due to the postgres volume mapping. If you wish to start over, simply delete your `.data/neosync-postgres` folder to do so. + +## Developing Neosync with Compose + +Check out the README at the root of the Neosync Github repository to learn more about how to development Neosync using compose and the dev-focused compose files. diff --git a/frontend/apps/web/app/api/auth/[...nextauth]/auth.ts b/frontend/apps/web/app/api/auth/[...nextauth]/auth.ts index 72f3b82d6..3bfc30f22 100644 --- a/frontend/apps/web/app/api/auth/[...nextauth]/auth.ts +++ b/frontend/apps/web/app/api/auth/[...nextauth]/auth.ts @@ -176,7 +176,6 @@ async function getTokenUrl(issuer: string): Promise { if (!oidcConfig.token_endpoint) { throw new Error('unable to find token endpoint'); } - console.log('found endpoint', oidcConfig.token_endpoint); return oidcConfig.token_endpoint; } catch (err) { console.log('failed', err);