Phase 2: Develop CI/CD for NPM/React projects

[jordanpadams]

💡 Description

Follow on to #67

Laying the groundwork for a future roundup requirement, let's create a new GitHub Action in a repo using NPM and test our what we want/need in order to support it.

For now, let's just build out the branch testing, unstable, and stable deliveries:

branch-cicd.yaml - install dependencies, build, verify success:

• https://github.com/NASA-PDS/wds-react#prerequisite
• https://github.com/NASA-PDS/wds-react#build-the-packages

unstable-cicd.yaml - install dependencies, build, deploy to dev npm (NOTE: requires package.json to already be updated as indicated in README):

• https://github.com/NASA-PDS/wds-react#prerequisite
• https://github.com/NASA-PDS/wds-react#build-the-packages
• npm publish --tag beta

stable-cicd.yaml - install dependencies, build, deploy to dev npm (NOTE: requires package.json to already be updated as indicated in README):

• https://github.com/NASA-PDS/wds-react#prerequisite
• https://github.com/NASA-PDS/wds-react#build-the-packages
• npm publish

Sub-tasks

• Develop Initial Architecture and Action Flow for React CI/CD Deployments #66
• Develop React/NodeJS template repo #61
• Develop Unstable GitHub Action CI #63
• Develop Stable GitHub Action CI #64
• Create workflows to utilize new roundup unstable/stable CI #73
• Design Continuous Deployment for React apps to AWS venues #74
• Develop Unstable GitHub Action CD to Dev venue #71
• Develop Staging GitHub Action CD to deploy to Test venue #72
• Develop Stable GitHub Action CD to Deploy to Prod venue #75
• Develop Branch Testing Github Action #62
• Develop PR Review GitHub Action CI/CD to Developer Staging venue #65
• Setup NPM Audits as part of CI on React projects wds-react-legacy#95

[nutjob4life]

Quick question:

In this comment @eddiesarevalo suggests using npm publish --tag unstable, but @jordanpadams writes (in the issue description ↑) npm publish --tag beta.

I have no feelings either way. What would you prefer? @anilnatha, do you have a preference?

[nutjob4life]

I believe this pull request will close the following issues

• Develop Unstable GitHub Action CI #63
• Develop Stable GitHub Action CI #64
• Develop Initial Architecture and Action Flow for React CI/CD Deployments #66
• Phase 2: Develop CI/CD for NPM/React projects #69

[jordanpadams]

📆 03/2024 status: In work. On schedule

[jordanpadams]

📆 04/2024 status: Updated end date to better reflect the work needed to complete continuous deployment for these applications. No impact on delivery.

[jordanpadams]

📆 05/2024 status: In work. On track for 6/20 delivery.

[nutjob4life]

This epic talks about using GitHub Actions for CI and CD.

CI Part

I've indentified the following repositories as using Node.js (based on presence of the package.json file). These repositories need the Roundup's newest support for Node packaging and artifact hosting plus branch testing—the "CI" part (2 open PRs):

• archive-viewer
• atlas
• doi-ui
• s3-browser-awssdk
• s3-browser-cloudfront
• template-repo-nodejs
• wds
• wds-react
• wds-react-legacy

CD Part

These repositories are using Terraform (based on the presence of .tf files)—the "CD" part:

• data-upload-manager
• nucleus
• pds-thin-egress-app
• registry-api
• registry-ref-data
• registry-sweepers
• updart

Overlap

Note: there is no overlap.

The Node.js projects don't have Terraform files, and the Terraform projects aren't Node.js.

Question: do you want me to provide the GitHub Actions for "CD" for the Node.js projects even though they don't have any Terraform files, or provide the GitHub Actions for "CD" for the Terraform projects even though they're not Node.js? Or both?

[nutjob4life]

@nutjob4life to query @anilnatha which of the above "CI" repositories is ripest for CD and to provide the Terraform files + the GitHub Action to run them for CD.

[nutjob4life]

@anilnatha consider this your official query: of the repositories under "CI Part" in this comment, which one would you like me to ① create some Terraform files for and ② add GitHub Actions to use those Terraform files to make an automatic dev-tier deployment?

[anilnatha]

@nutjob4life None of the wds-* repos require terraform, and atlas is the repo for the PDS IMG Atlas Product Search tool that is hosted by PDS IMG so don't need to create terraform for it either. I can't speak for the other repos.

[anilnatha]

I would imagine that we need terraform for the s3-browser-* repos. 🤷🏽

@eddiesarevalo ?

[jordanpadams]

@anilnatha to rephrase the question, if we wanted to be able to view and/or test what you and Eddie are currently working on, what repository would benefit from continuous deployment and/or containerization? Note: This may change in the future once we get wordpress in the loop.

[anilnatha]

I think it's in our best interest to focus on flushing out the cloud architecture I initially proposed and building out the portal-wp repo. We don't have to have the wordpress architecture solidified right now, but if we could at least resolve the issues with building out the portals frontend containers and getting those served through CI/CD, it would be a big step forward.

[nutjob4life]

Okay so is the idea to make portal-wp be like a template repo that you duplicate so you can stand-up Wordpress sites easily or more as just a "model", as in, "here's how you'd do it as a fully-fleshed example"?

[jordanpadams]

@nutjob4life this will be the one and only. we don't care about other wordpress sites and how to deploy this. this can be as application specific as needed.

[tloubrieu-jpl]

The build is broken, Sean will try to fix it since @anilnatha is busy and he wants to learn that.

[anilnatha]

@nutjob4life The icons do still build when the npm run build-icons command is issued. The issue you encountered is something that has been plaguing us since Eddie first implemented the icon build. We haven't been able to resolve the issue yet. :( If you look in the distribution folder, you'll find that the icons do in fact get rendered correctly [cjs|esm]/components/icons/

CC: @eddiesarevalo

[anilnatha]

@nutjob4life I created a ticket to track the issue you reported, but it shouldn't be a blocker. Let me know if you are in fact blocked though so I can prioritize the bug fix. Reference: NASA-PDS/wds-react#52

[anilnatha]

@nutjob4life I fixed the build-icons issue, PR to get that reviewed and merged is here.

[nutjob4life]

@anilnatha getting closer! I'm not up to the docker compose up step:

 => => resolve docker.io/library/wordpress:6.3.1-php8.2-apache@sha256:34c77a88700b856ad2271e386de6214fba4088dfc5b94  0.0s
 => => sha256:61c2e8cf23ab24df88a251a33d11d42677dca7e4eba0a028c1838d6f39c44843 4.71kB / 4.71kB                       0.0s
 => => sha256:34c77a88700b856ad2271e386de6214fba4088dfc5b944067aafebad8a8f86b1 1.86kB / 1.86kB                       0.0s
 => => sha256:53be2feed7f7f8b2451fe62d7bf250b0d7455eed5ac436afe2f57e3b23020e00 18.37kB / 18.37kB                     0.0s
 => [backend internal] load build context                                                                            0.0s
 => => transferring context: 2B                                                                                      0.0s
 => CACHED [backend 2/6] WORKDIR /usr/src/wordpress/wp-content                                                       0.0s
 => ERROR [backend 3/6] COPY ./data/wp-content/themes/ themes/                                                       0.0s
 => ERROR [backend 4/6] COPY ./data/wp-content/plugins/ plugins/                                                     0.0s
------
 > [backend 3/6] COPY ./data/wp-content/themes/ themes/:
------
------
 > [backend 4/6] COPY ./data/wp-content/plugins/ plugins/:
------
failed to solve: failed to compute cache key: failed to calculate checksum of ref 68b7b2b3-558e-4f9e-9818-1541749fbbb3::7byfdhiihsb66bojnnpvq6mmx: failed to walk /var/lib/docker/tmp/buildkit-mount3244437086/data/wp-content: lstat /var/lib/docker/tmp/buildkit-mount3244437086/data/wp-content: no such file or directory

What do you think?

[anilnatha]

@nutjob4life With where we are with the MVP I think we should skip trying to get the WordPress set up and focus on just getting the frontend application up and running. We will be starting on the wordpress set up soon, but it's not needed for the MVP.

Should I comment out the wordpress configuration(s)

[nutjob4life]

@anilnatha I was just following the instructions in the README, I swear! 😇

[nutjob4life]

@jordanpadams @tloubrieu-jpl I know we're trying to reduce the S3 buckets in use, but in order to turn this from a "technology demonstration" to an actual hosted app, I'll need a bucket. What's the process for procuring one?

[jordanpadams]

@nutjob4life I have no problem with all the buckets we need, just not stale/unused/undocumented buckets :-)

@viviant100 @sjoshi-jpl how should we put requests in for this? If this is in MCP-Dev, is there an FAQ somewhere we can direct users when they ask "How can I create an S3 bucket?"

[nutjob4life]

@jordanpadams copy that! 😁

[jordanpadams]

@sjoshi-jpl @viviant100 @tloubrieu-jpl ping! can we get a bucket created ASAP in Dev? This is both for @nutjob4life testing, as well as @eddiesarevalo to deploy manually for testing

[nutjob4life]

@viviant100 is fast 🏎️💨

The bucket was just created, s3://pds-portal-demo … thanks much!

[nutjob4life]

When running terraform apply:

arn:aws:sts::441083951559:assumed-role/mcp-tenantDeveloper/skelly is not authorized to perform: s3:PutBucketPolicy on resource: "arn:aws:s3:::pds-portal-demo" because public policies are blocked by the BlockPublicPolicy block public access setting

This is using MCP credentials from "Kion":

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_SESSION_TOKEN

[jordanpadams]

📆 08/2024 status: Work continues. On schedule to have a deployed NPM project by end of sprint.

[tloubrieu-jpl]

@sjoshi-jpl is helping on that.

Developers can not PUT to s3 buckets. We will need support from SAs on that.

[nutjob4life]

Working "by hand", finally have first success using terraform apply to get files to appear in S3!

In order to still make a "proof of concept", I'll work next to see if I can hook up CloudFront. Meanwhile work on OIDC continues in parallel.

[nutjob4life]

CloudFront update:

I'm not exactly sure what's wrong. In CloudFront instance E18VQ4K51WT0LV, the origin s3_webmod_bkt is connected to pds-portal-demo.s3.us-west-2.amazonaws.com and I've added origin path /portal.

But visiting https://pds-sit.mcp.nasa.gov/portal gives

<Error>
  <Code>AccessDenied</Code>
  <Message>Access Denied</Message>
  <RequestId>2RZ44QYTN6MSYYXB</RequestId>
  <HostId>
    tqqqiPJRIWD96jInhV7IcHLzl0KpPy3MD+GjnUMoRh2XU9zVHb/ELh2KA3X0bFCGI1MzYkMtYe8=
  </HostId>
</Error>

Gonna need some AWS experts to chime in 🙏 @viviant100 @sjoshi-jpl @tloubrieu-jpl

[nutjob4life]

Thanks to @viviant100 in this thread I can get closer—but thanks to JPL Cyber Security I can't actually access VPN (or even public .jpl.nasa.gov sites), so will put this on hold until they unblock me.

[nutjob4life]

Okay, once again kudos to @viviant100 for getting things configured. I'm happy to report that visiting https://pds-sit.mcp.nasa.gov/portal/ (note you must use the trailing /) finally gives the expected portal-wp web app. Of course, this is still done using the hand-run terraform apply, and it's using an older version of the app.

Next steps are to ① use a more recent update to the develop tree of portal-wp; ② get the OIDC authentication working so GitHub Actions can do this; ③ consider production deployment scenarios; ④ foresee the unforeseen issues.

[nutjob4life]

@anilnatha @eddiesarevalo I've updated my local wds, wds-react, and portal-wp directories from GitHub with the latest develop branch commits on each.

In wds, I did nvm use && npm clean-install && npm run build:css && npm run build:icons && npm link. So far so good.

In wds-react, I did nvm use && npm clean-install && npm link && npm link @nasapds/wds && npm run build-icons. Also works.

In portal-wp/apps/frontend, though, is where I run into some trouble. After doing nvm use && npm clean-install && npm link && npm link @nasapds/wds && npm link @nasapds/wds-react && npm run build, the run build steps fails with:

> pds-portal-frontend@0.0.0 build
> tsc && vite build

src/pages/search/index.tsx:35:3 - error TS2305: Module '"@nasapds/wds-react"' has no exported member 'Button'.

35   Button,
     ~~~~~~

src/pages/search/index.tsx:36:3 - error TS2305: Module '"@nasapds/wds-react"' has no exported member 'FeaturedLink'.

36   FeaturedLink,
     ~~~~~~~~~~~~

src/pages/search/index.tsx:37:3 - error TS2305: Module '"@nasapds/wds-react"' has no exported member 'FeaturedLinkDetails'.

37   FeaturedLinkDetails,
     ~~~~~~~~~~~~~~~~~~~

src/pages/search/index.tsx:38:3 - error TS2305: Module '"@nasapds/wds-react"' has no exported member 'FeaturedLinkDetailsVariant'.

38   FeaturedLinkDetailsVariant,
     ~~~~~~~~~~~~~~~~~~~~~~~~~~

src/pages/search/index.tsx:700:21 - error TS2322: Type '"search"' is not assignable to type '"filled" | "outlined" | "standard" | undefined'.

700                     variant="search"
                        ~~~~~~~

  ../../../wds-react/node_modules/@mui/material/TextField/TextField.d.ts:201:3
    201   variant: 'filled';
          ~~~~~~~
    The expected type comes from property 'variant' which is declared here on type 'IntrinsicAttributes & TextFieldProps'


Found 5 errors in the same file, starting at: src/pages/search/index.tsx:35

Ideas? Thanks in advance!

[nutjob4life]

Thanks @eddiesarevalo for your help 🎉

The next issue now is: when you visit https://pds-sit.mcp.nasa.gov/portal/ the line <script type="module" crossorigin src="/assets/index-Cqbnc77p.js"></script> in index.html tries to load https://pds-sit.mcp.nasa.gov/assets/index-Cqbnc77p.js instead of https://pds-sit.mcp.nasa.gov/portal/assets/index-Cqbnc77p.js with /portal/ in it

This was working before, so I think something might've changed

[tloubrieu-jpl]

Waiting for @sjoshi-jpl regarding the status of the ODC authentication. Will see that next week.

[jordanpadams]

📆 09/2024 status: delayed several sprints working out OIDC privileges. operations task. no impact on delivery.

[jordanpadams]

Calling this phase completed. Moving remaining sub-tasks to B15.1 to work with Drupal CICD