Next.js Initial Migration

[jonkafton]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/5244

Description (What does it do?)

Migrates the bulk of the site onto Next.js with the intention of identifying the areas of work remaining to get this to a full release and establishing viability of continuing feature development on the main branch while this is in progress.

The aim for the first Next.js release is feature parity with the existing site. The code will have been migrated onto Next.js and the base HTML for each route server side rendered with all head and metadata content. The React components that make up the body are generally Client Components on Next.js, which will determine whether to render them on the server or the client depending on whether they need to make API calls to hydrate their content or have client side interactivity.

The Next.js app is added at ./frontends/main. The existing entrypoints, ./frontends/mit-learn remains in the repository, with its contents moved into ./frontends/main as they are migrated to run on Next.js. This helps ensure Git treats migrated files as moves so we can merge new features from main without much difficulty.

The general outcome of this initial migration was that the existing site can be migrated onto Next.js with relatively minimal code change to the React components and achieve a similar end user experience, plus the page metadata rendered on the server for search and social media previews. We should be able to continue feature development on the main branch while we ready this one for release, given there is little divergence in the components themselves and we can merge in without too much headache.

After a first release we can progressively optimize by opting Client Components into pre-rendering by fetching data on the server and loading it into the React Query cache.

The main area of work to ready this for production will be in reinstating unit tests that are not compatible with Next.js' routing (potentially replacing them with browser tests), testing the site for regressions and applying any fixes and putting the build pipelines and hosting in place. There are new paradigms introduced and plenty of gotchas around the demarcation of server and client components, when things are server rendered, and how rendered content is serialized for hydration in the browser.

Included in this PR:

• Next.js project scaffolding.

• App Router filesystem based routing providing entry points to our page components. This replaces React Router. The page entry points are Server Components and apply any metadata and can make API calls to populate the React Query cache.

• Migrate components to run on Next.js. This involves replacing any React Router routing or navigation with Next.js equivalents, plus replacing images to make use of Next.js image optimzation. These are migrated:

  • The majority of pages, excluding some editor pages. The pages are marked as Client Components ("use client" directive). Dependency page-components, components and common code are also migrated.

  • ol-components are Client Components by nature and have been updated to for Next.js navigation.

• Metadata utility for producing title, description and social media tags on the server. Resource data fetched on server to populate for pages with an open drawer.

• Example of prefetching API data on the server and populating the React Query cached for a fully server side rendered homepage carousel.

• Env vars updated for the Next.js build. Values for the client must be prefixed and accessed with process.env.NEXT_PUBLIC_<VAR>.

• Images in use migrated over and folders cleaned up.

Not included in this PR:

This constitutes the work remaining, some of which we'll need to determine the best approach for and itemize.

• Unit Tests - Any of our tests that run React Router will need to be replaced (renderTestApp() and renderRoutesWithProviders()). We may take an approach where we mock Next.js' App Router, investigate solutions idiomatic to Next.js or replace these with Playwright browser tests. The remainder that test components directly or outside of the context of React Router should be able to be reinstated without too much trouble by mocking out next/link and useRouter() from next/navigation.

• Regression Testing - We will need to test the migration manually for any regression and raise bug tickets accordingly.

• RestrictedRoute logic - This depends on the user session and therefore the wide decision on whether we want to pre-render use content. Likely not for initial release - we'll need to move RestrictedRoute to their Client Component pages to work independently of routing.

• Migrate remaining pages - Article detail, create, edit, upsert pages, edit channel page.

• Remaining metadata - Apply metadata to remaining pages. Some of these will need server side API calls to populate.

• Images - Next.js provides image optimization and mechanisms to prevent page jump as imaged are presented, with the caveat that the dimensions must be know. We need to establish whether we should use this in place of Embedly, its utility for background images and then make the updates across remaining images. One issue is that image URLs must be whitelisted in the Next.js config - a problem given we are ingesting from many sources and these are likely unknown.

• Posthog - Needs to be migrated onto the App Router](https://posthog.com/docs/libraries/next-js).

• Environment variables - Public variable for the browser must be prefixed NEXT_PUBLIC_ and referenced in the code on process.env for substitution at build time. We need to replace any remaining occurrences of APP_SETTINGS and move the .env file alongside our existing ./env/*.env.

• Build config - Includes adding a Dockerfile for watch mode development and for running in production.

• CI Pipeline - Configure a CI job to build the docker image and potentially push to a container registry.

Operations items outside of repo

• Hosting - To be determined, but likely running on Heroku's docker runtime initially.

• CDN - If we are only server rendering public content, all routes can be cached, with caution. The cache will need to miss for cached paths at different search params as we populate page metadata dependent on open resource drawer.

• Monitoring and alerting - Operations controls in place for visibility of an additional service. For public heavily cached content we're relatively immune to spikes in traffic. If we pre-render user content we'll need to size and scale according to load.

Post initial release

• Phase 2 - The subsequent phase 2 will involve optimizing for pre-rendered public content for optimal page load, server actions and caching and looking into partial pre-rendering for streaming user content in parallel to the initial render.

How can this be tested?

• Copy ./frontends/main/.env.example to ./frontends/main/.env.
• yarn install
• Start backend services, docker compose up nginx web.
• Stop the watch container, if running.
• In ./frontends/main run PORT=8062 yarn dev
• Navigate to http://od.odl.local:8062 (usual assumptions on local DNS)

[jonkafton]

Replaced by #1505 so we can use nextjs as a feature branch and PR against that.