chore: put server assets on CDN #9278

mattkrick · 2023-12-06T23:09:25Z

Description

Publicly host server assets so they can be used by downstream services like email, mattermost, etc.

fix #9111
fix #8553

Demo

Testing scenarios

in dev, with no CDN

http://localhost:3000/jira-attachments/foo.png results in a image not found .png
emails using anonymous avatar work correctly

in prod, with no CDN (after running preDeploy)

http://localhost:3000/jira-attachments/foo.png results in a image not found .png
notifications published to mattermost include an absolute url starting with env.HOST
images in manifest.json are available
emails using anonymous avatar work correctly

in prod, with CDN (after running preDeploy)

http://localhost:3000/jira-attachments/foo.png results in a image not found .png
notifications published to mattermost include an absolute url starting with CDN_BASE_URL
images in manifest.json are available
templates upload to the existing location

Signed-off-by: Matt Krick <matt.krick@gmail.com>

mattkrick · 2023-12-13T00:07:45Z

packages/server/initPublicPath.ts

@@ -0,0 +1,16 @@
+import appOrigin from './appOrigin'
+
+declare let __webpack_public_path__: string


we can't set the public path at buildtime because different PPMIs will host their assets in different places.
since the public path is defined during predeploy, we must set it at runtime

mattkrick · 2023-12-13T00:09:04Z

packages/server/jiraImagesHandler.ts

-      path.join(__dirname, jiraPlaceholder.slice(__webpack_public_path__.length))
-    )
+    try {
+      const res = await fetch(jiraPlaceholder)


all other assets need a public URL except this one, so instead of special casing this one I just turned it into a fetch vs. a file system read.

mattkrick · 2023-12-13T00:10:36Z

packages/server/utils/serveStatic.ts

-  [path.join(PROJECT_ROOT, 'static')]: !__PRODUCTION__,
-  [path.join(PROJECT_ROOT, 'dev', 'dll')]: !__PRODUCTION__
+  // publish server assets at /static
+  [path.join(PROJECT_ROOT, 'dist')]: __PRODUCTION__,


path is left intact, e.g. dist/images/imageNotFound.png will be published at /static/images/imageNotFound.png

mattkrick · 2023-12-13T00:12:09Z

scripts/toolboxSrc/applyEnvVarsToClientAssets.ts

@@ -82,7 +78,7 @@ const rewriteIndexHTML = () => {
  const keys = `<script>window.__ACTION__=${JSON.stringify(clientKeys)}</script>`
  const rawHTML = skeleton
    .replace('<head>', `<head>${noindex}${keys}`)
-    .replaceAll('__PUBLIC_PATH__', getCDNURL())
+    .replaceAll('__PUBLIC_PATH__', __webpack_public_path__.replace(/\/$/, ''))


remove trailing slash

mattkrick · 2023-12-13T00:13:02Z

scripts/toolboxSrc/pushToCDN.ts

@@ -2,6 +2,11 @@ import fs from 'fs'
 import getFileStoreManager from 'parabol-server/fileStorage/getFileStoreManager'
 import path from 'path'
 import getProjectRoot from '../webpack/utils/getProjectRoot'
+;(require as any).context(


since the DB needs these files, we require them here so they get added to the server bundle, e.g. dist/images/templates

mattkrick · 2023-12-13T00:14:46Z

scripts/toolboxSrc/pushToCDN.ts

@@ -25,46 +30,78 @@ const pushClientAssetsToCDN = async () => {
  console.log(`⛅️ Uploaded ${dirEnts.length} client assets to CDN`)
 }

-const pushTemplatesToCDN = async () => {
+const pushServerAssetsToCDN = async () => {


we were just pushing client assets, now we push server assets, too.
This is useful for things like:

anonymousAvatar getting referenced in emails

our logo's URL getting published in mattermost messages

mattkrick · 2023-12-13T00:16:13Z

scripts/webpack/dev.servers.config.js

@@ -25,8 +26,8 @@ module.exports = {
    __dirname: false
  },
  entry: {
-    web: [DOTENV, path.join(SERVER_ROOT, 'server.ts')],
-    gqlExecutor: [DOTENV, path.join(GQL_ROOT, 'gqlExecutor.ts')]
+    web: [DOTENV, INIT_PUBLIC_PATH, path.join(SERVER_ROOT, 'server.ts')],


the public path has to get set before imports use it, so set it here because that way the import order of server.ts doesn't matter

Signed-off-by: Matt Krick <matt.krick@gmail.com>

.github/workflows/test.yml

mattkrick · 2023-12-14T00:16:01Z

.github/workflows/test.yml

@@ -129,5 +130,5 @@ jobs:
        uses: actions/upload-artifact@v2
        with:
          name: test-results
-          path: test-results/
+          path: packages/integration-tests/test-results/


this has been wrong since forever, i just didn't ever need it until today 😞

mattkrick · 2024-01-24T20:21:41Z

@igor would you mind giving this a review?

igorlesnenko

I do: yarn build, yarn predeploy, yarn start
And images of meetings are not displayed. Am I missing something?

igorlesnenko · 2024-01-25T04:03:45Z

scripts/toolboxSrc/applyEnvVarsToClientAssets.ts

@@ -47,8 +45,6 @@ const writeManifest = () => {
  const manifestPath = path.join(clientDir, 'manifest.json')
  fs.writeFileSync(manifestPath, JSON.stringify(manifest))
  // move the referenced icons into the client build


Not moving already

Signed-off-by: Matt Krick <matt.krick@gmail.com>

chore: put server assets on CDN

30feb15

Signed-off-by: Matt Krick <matt.krick@gmail.com>

github-actions bot added the size/m label Dec 6, 2023

mattkrick added 6 commits December 8, 2023 13:24

Merge branch 'master' into fix/9111/jira-4xx

faa04e5

Merge branch 'master' into fix/9111/jira-4xx

24fdbc8

Signed-off-by: Matt Krick <matt.krick@gmail.com>

fix: stray uncorked res

f587f48

Signed-off-by: Matt Krick <matt.krick@gmail.com>

Merge branch 'master' into fix/9111/jira-4xx

f5f7bf1

Signed-off-by: Matt Krick <matt.krick@gmail.com>

prod support for public path

c91a5e7

Signed-off-by: Matt Krick <matt.krick@gmail.com>

support nested dirs

50e8377

Signed-off-by: Matt Krick <matt.krick@gmail.com>

mattkrick commented Dec 13, 2023

View reviewed changes

mattkrick requested a review from BartoszJarocki December 13, 2023 00:19

mattkrick added 2 commits December 13, 2023 10:34

log bg processes

7aadd68

Signed-off-by: Matt Krick <matt.krick@gmail.com>

fix failed test path

a6ebe92

Signed-off-by: Matt Krick <matt.krick@gmail.com>

mattkrick removed the request for review from BartoszJarocki December 13, 2023 22:28

mattkrick marked this pull request as draft December 13, 2023 22:28

mattkrick added 2 commits December 13, 2023 15:20

simplify webpack build

a84d701

Signed-off-by: Matt Krick <matt.krick@gmail.com>

change proto to http for CI localhost

6a63113

Signed-off-by: Matt Krick <matt.krick@gmail.com>

mattkrick commented Dec 14, 2023

View reviewed changes

.github/workflows/test.yml Show resolved Hide resolved

mattkrick commented Dec 14, 2023

View reviewed changes

mattkrick marked this pull request as ready for review December 14, 2023 00:16

mattkrick requested a review from BartoszJarocki December 14, 2023 00:16

mattkrick requested review from igorlesnenko and removed request for BartoszJarocki January 24, 2024 20:19

igorlesnenko reviewed Jan 25, 2024

View reviewed changes

fix: remove comment

90b96b3

Signed-off-by: Matt Krick <matt.krick@gmail.com>

mattkrick requested a review from Dschoordsch March 2, 2024 01:55

Dschoordsch approved these changes Mar 4, 2024

View reviewed changes

mattkrick added 2 commits March 5, 2024 09:40

Merge branch 'master' into fix/9111/jira-4xx

e869495

Signed-off-by: Matt Krick <matt.krick@gmail.com>

add embedder back to build

b8ad9f0

Signed-off-by: Matt Krick <matt.krick@gmail.com>

mattkrick merged commit 06c1f7e into master Mar 5, 2024
4 checks passed

mattkrick deleted the fix/9111/jira-4xx branch March 5, 2024 23:25

parabol-release-bot bot mentioned this pull request Mar 5, 2024

chore(release): release v7.21.0 #9505

Merged

github-actions bot mentioned this pull request Mar 6, 2024

chore(release): Test v7.21.0 #9510

Closed

24 tasks

mattkrick mentioned this pull request Mar 14, 2024

fix: node-loader that ignores public path #9537

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: put server assets on CDN #9278

chore: put server assets on CDN #9278

mattkrick commented Dec 6, 2023 •

edited

mattkrick Dec 13, 2023

mattkrick Dec 13, 2023

mattkrick Dec 13, 2023

mattkrick Dec 13, 2023

mattkrick Dec 13, 2023

mattkrick Dec 13, 2023

mattkrick Dec 13, 2023

mattkrick Dec 14, 2023

mattkrick commented Jan 24, 2024

igorlesnenko left a comment

igorlesnenko Jan 25, 2024

		@@ -0,0 +1,16 @@
		import appOrigin from './appOrigin'

		declare let __webpack_public_path__: string

chore: put server assets on CDN #9278

chore: put server assets on CDN #9278

Conversation

mattkrick commented Dec 6, 2023 • edited

Description

Demo

Testing scenarios

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattkrick commented Jan 24, 2024

igorlesnenko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattkrick commented Dec 6, 2023 •

edited