-
Notifications
You must be signed in to change notification settings - Fork 22
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fslogical: Initial support for Firestore
This change adds initial support for importing document collections from Google Cloud Firestore. The new dialect operates in a key-pagination mode for initial backfills and then switches to a streaming mode once it has caught up. The implementation has several caveats: * All documents to be synched must have an updated_at field (the name is configurable) which is set to the server-provided timestamp. These document timestamps are used to establish a consistent-point barrier that separates backfills from streaming operations. * Document deletions that occur outside of streaming mode may be lost. There does not appear to be a service in Firestore which is analogous to a transaction log or CDC feed. Testing uses a custom container image, uploaded to the GitHub package repo, that contains a local firestore emulator. We do not depend on production Firestore. The emulator image is built and deployed by a new workflow.
- Loading branch information
Showing
20 changed files
with
1,520 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # Build from a Node base, add the required JDK, then install. | ||
| # https://firebase.google.com/docs/emulator-suite/install_and_configure | ||
| FROM node:alpine | ||
| RUN apk add openjdk11 | ||
| RUN npm install -g firebase-tools | ||
| # Pre-cache the firestore emulator. | ||
| RUN firebase setup:emulators:firestore | ||
| COPY emulators.json . | ||
| CMD ["firebase", "emulators:start", "--only", "firestore", "-c", "emulators.json"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| This directory builds a container which runs the Google Cloud Firestore emulator. We save time in | ||
| each of the tests by pre-caching the jar files which the firebase cli will download. We also | ||
| configure the emulator to bind to all addresses instead of loopback. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| { | ||
| "emulators": { | ||
| "firestore": { | ||
| "host": "0.0.0.0" | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Build the Firestore emulator docker image and push it to the GitHub | ||
| # package repository. | ||
| name: Firestore Emulator | ||
| permissions: | ||
| contents: read | ||
| packages: write | ||
| on: | ||
| push: | ||
| branches: [ master ] | ||
| paths: | ||
| - ".github/firestore/**" | ||
| pull_request: | ||
| paths: | ||
| - ".github/firestore/**" | ||
|
|
||
| env: | ||
| REGISTRY: ghcr.io | ||
| IMAGE_NAME: ghcr.io/${{ github.repository }}/firestore-emulator | ||
|
|
||
| jobs: | ||
| # https://docs.github.com/en/actions/publishing-packages/publishing-docker-images#publishing-images-to-github-packages | ||
| push: | ||
| name: Push Firestore Emulator to GH Packages | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Check out the repo | ||
| uses: actions/checkout@v3 | ||
|
|
||
| - name: Log in to GitHub Package Registry | ||
| uses: docker/login-action@v2 | ||
| with: | ||
| registry: ${{ env.REGISTRY }} | ||
| username: ${{ github.actor }} | ||
| password: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
||
| - name: Extract metadata (tags, labels) for Docker | ||
| id: meta | ||
| uses: docker/metadata-action@v4 | ||
| with: | ||
| images: ${{ env.IMAGE_NAME }} | ||
|
|
||
| - name: Build and push Docker image | ||
| uses: docker/build-push-action@v3 | ||
| with: | ||
| context: ./.github/firestore | ||
| push: true | ||
| tags: ${{ env.IMAGE_NAME }}:latest | ||
| labels: ${{ steps.meta.outputs.labels }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.