Create app backend with PG support & E2E Test #1726

shmidt-i · 2020-07-23T20:14:06Z

Prelude:

There's been some discussions around approaches for the DB setup for the backend, e.g. #1531, #1527 and #1598.
While developing, we are using in-memory instance of SQLite, which allows us to iterate fast. But, on the same time we lack checks that stuff actually works together with PostgreSQL (which we plan to use for our demo deployment https://github.com/spotify/backstage/milestone/14 ), as a result - #1719 .
This is a step forward, it doesn't solve all the questions regarding production DB setup, but allows us to develop in a more safer way.

This PR introduces:

Backend package is now part of the backstage-cli create-app (catalog, scaffolder, auth, identity and tech-docs plugins included as of now)
Both packages/example-backend and backend created through cli have support for the PostgreSQL db.
CLI e2e test now includes check for the correct creation and startup of the backend with PostgreSQL db (migrations included).

How to proceed:

See https://github.com/spotify/backstage/pull/1726/checks?check_run_id=905018391
This is exactly error from Backend migrations fail on Postgres for ~v0.1.1-alpha.15 #1719, replicated in the github-actions environment.
Probably easier to go commit by commit
Merge fix Catalog: fix migrations for PG #1729 first (✅)
Check that new build succeeds (✅)
Merge this PR
Profit 😎

freben · 2020-07-23T20:24:12Z

.github/workflows/cli.yml

@@ -13,6 +13,18 @@ jobs:
  build:
    runs-on: ${{ matrix.os }}

+    services:
+      postgres:
+        image: postgres:10.8


Why this one specifically? There are less precise (and newer?) ones too right

Wow you are fast 😎
Mostly created this PR to get the action runs. But feel free to raise things to improve ;)
Here - yes, definitely can go for newer and less precise version

.github/workflows/cli.yml

freben · 2020-07-23T20:28:43Z

packages/backend/package.json

@@ -34,6 +34,7 @@
    "dockerode": "^3.2.0",
    "express": "^4.17.1",
    "knex": "^0.21.1",
+    "pg": "^8.3.0",


8 here, 10 in the image - is that expected?

This is npm package, and before is docker image, idk if think they are THAT correlated?

Me neither, just checking

Checked, they are not correlated, all good

packages/backend/src/index.ts

shmidt-i · 2020-07-24T08:12:05Z

packages/cli/templates/default-app/packages/backend/README.md

+AUTH_GITHUB_CLIENT_ID=x AUTH_GITHUB_CLIENT_SECRET=x \
+AUTH_OAUTH2_CLIENT_ID=x AUTH_OAUTH2_CLIENT_SECRET=x \
+AUTH_OAUTH2_AUTH_URL=x AUTH_OAUTH2_TOKEN_URL=x \
+ROLLBAR_ACCOUNT_TOKEN=x \


shmidt-i · 2020-07-24T08:13:16Z

packages/cli/templates/default-app/packages/backend/package.json.hbs

+    "@backstage/backend-common": "^{{version}}",
+    "@backstage/catalog-model": "^{{version}}",
+    "@backstage/config": "^0.1.1-alpha.13",
+    "@backstage/config-loader": "^0.1.1-alpha.13",


@Rugvip do you know why this works only for this particular version and not if I use "^{{version}}" here?

Yep, needs #1734

shmidt-i · 2020-07-24T08:23:28Z

packages/cli/e2e-test/cli-e2e-test.js

@@ -16,6 +16,7 @@

 const os = require('os');
 const fs = require('fs-extra');
+const fetch = require('node-fetch');


I saw whatwg-fetch in the repo, should we use that @benjaminr ?

Apologies @benjaminr , I meant @benjdlambert 🙃

yeah I was moving more towards using that as it's a one package that can polyfill frontend and backend stuff.

also it plays nicely with msw too. but it needs to be version v2 as v3 causes issues as the react-lazylog depends on a different version which causes issues when compiling the backend because of package hoisting,

Looked into it and it seems like it's actually not both backend and frontend at all :)
And not planning to😜
JakeChampion/fetch#184

shmidt-i · 2020-07-24T08:31:57Z

packages/cli/e2e-test/cli-e2e-test.js

+  try {
+    await waitFor(() => stdout.includes('Listening on ') || stderr !== '');
+    if (stderr !== '') {
+      // Skipping the whole block


Print the stderr

shmidt-i · 2020-07-24T08:38:29Z

packages/cli/e2e-test/helpers.js

+ * Checks every 100ms
+ * .cancel() is available
+ * @returns {Promise} Promise of resolution
+ */


Remove the comment

Rugvip · 2020-07-24T09:13:26Z

packages/backend/src/index.ts

+          port: process.env.POSTGRES_PORT,
+          host: process.env.POSTGRES_HOST,
+          user: process.env.POSTGRES_USER,
+          password: process.env.POSTGRES_PASSWORD,


Can we grab this from config? 😅

Should we? I mean secrets in the config, idk, is this what we decided to do?

The config loader has support for secrets from multiple sources, so it's more flexible than this. You can load it from files and whatnot too.

packages/cli/e2e-test/cli-e2e-test.js

packages/cli/templates/default-app/.eslintrc.js

packages/cli/templates/default-app/packages/backend/src/index.ts.hbs

Rugvip · 2020-07-24T09:24:00Z

packages/cli/templates/default-app/packages/backend/src/plugins/identity.ts

+import { PluginEnvironment } from '../types';
+
+export default async function createPlugin({ logger }: PluginEnvironment) {
+  return await createRouter({ logger });


Wonder if we should start exporting default factories from backend plugins to remove the need for all of this extra wiring in the backend. Kinda a 90% use-case coverage thing. It'll likely also make a config-only deployment easier.

Not in this PR though :p

Rugvip · 2020-07-24T09:26:36Z

packages/cli/templates/default-app/tsconfig.json

+    "packages/*/src",
+    "plugins/*/src",
+    "plugins/*/dev",
+    "plugins/*/migrations"


…reate-app-backend

Rugvip

shmidt-i added 2 commits July 23, 2020 22:10

feat(cli): create-app creates backend as well with PG support

2893daf

feat(backend): add support for PG

189e1bf

shmidt-i force-pushed the shmidt-i/create-app-backend branch 2 times, most recently from f0f119c to cf1250e Compare July 23, 2020 20:25

freben reviewed Jul 23, 2020

View reviewed changes

shmidt-i force-pushed the shmidt-i/create-app-backend branch 5 times, most recently from 124e7ed to 43211cb Compare July 24, 2020 01:00

feat(cli): add e2e test for backend

5f02544

shmidt-i force-pushed the shmidt-i/create-app-backend branch from 43211cb to 5f02544 Compare July 24, 2020 01:16

shmidt-i marked this pull request as ready for review July 24, 2020 02:05

shmidt-i requested a review from a team as a code owner July 24, 2020 02:05

fix: tsc and pr comments

a6b8602

shmidt-i force-pushed the shmidt-i/create-app-backend branch from 48c2933 to a6b8602 Compare July 24, 2020 07:43

shmidt-i commented Jul 24, 2020

View reviewed changes

Rugvip reviewed Jul 24, 2020

View reviewed changes

shmidt-i added 3 commits July 24, 2020 12:13

Merge branch 'master' of github.com:spotify/backstage into shmidt-i/c…

411b4af

…reate-app-backend

fix(cli): e2e backend PR adjustments

e5ed8f9

Merge branch 'master' of github.com:spotify/backstage into shmidt-i/c…

3f046ed

…reate-app-backend

Rugvip approved these changes Jul 27, 2020

View reviewed changes

shmidt-i merged commit 6edbb0c into master Jul 27, 2020

shmidt-i deleted the shmidt-i/create-app-backend branch July 27, 2020 14:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create app backend with PG support & E2E Test #1726

Create app backend with PG support & E2E Test #1726

shmidt-i commented Jul 23, 2020 •

edited

Loading

freben Jul 23, 2020

shmidt-i Jul 23, 2020

freben Jul 23, 2020

shmidt-i Jul 23, 2020

freben Jul 23, 2020

shmidt-i Jul 27, 2020

shmidt-i Jul 24, 2020

shmidt-i Jul 24, 2020

Rugvip Jul 24, 2020

shmidt-i Jul 24, 2020

shmidt-i Jul 24, 2020

benjdlambert Jul 27, 2020

shmidt-i Jul 27, 2020 •

edited

Loading

shmidt-i Jul 24, 2020

shmidt-i Jul 24, 2020

Rugvip Jul 24, 2020

shmidt-i Jul 24, 2020

Rugvip Jul 24, 2020

Rugvip Jul 24, 2020

Rugvip Jul 24, 2020

Rugvip left a comment

Create app backend with PG support & E2E Test #1726

Create app backend with PG support & E2E Test #1726

Conversation

shmidt-i commented Jul 23, 2020 • edited Loading

Prelude:

This PR introduces:

How to proceed:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmidt-i Jul 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rugvip left a comment

Choose a reason for hiding this comment

shmidt-i commented Jul 23, 2020 •

edited

Loading

shmidt-i Jul 27, 2020 •

edited

Loading