Dashboard Routing and Build Optimizations

[anthonylasso]

Description

nginx location routing is not gracefully handling next.js dynamic routing in a graceful and scalable way.
By using an internal node server and proxying dashboard requests to it, all next.js features are enabled without special configurations.

next.js output file tracing also helps reduce the overall featureform image size (~4.2GBs vs. ~1.9GBs).

More importantly, the production dashboard operates exactly as our local node development environments do. Currently unexpected errors can occur once our code hits production that we might miss in local dev.

Proposal Document
Excalidraw Design

This merge request will remain in draft for the next couple of weeks, to enable thorough testing + evaluation.

Testing Schedule:

• Local Mode
• Docker Mode
• K8S / Minikube Mode
• Bug Bash

Type of change

Routing architecture + docker build change.

Does this correspond to an open issue?

#774

Select type(s) of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist:

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have fixed any merge conflicts

[codecov]

Codecov Report

Merging #781 (3b9585c) into main (6f7b498) will decrease coverage by 2.93%.
The diff coverage is 54.87%.

@@            Coverage Diff             @@
##             main     #781      +/-   ##
==========================================
- Coverage   59.71%   56.78%   -2.93%     
==========================================
  Files          63       66       +3     
  Lines       12408    12974     +566     
==========================================
- Hits         7409     7367      -42     
- Misses       4232     4889     +657     
+ Partials      767      718      -49     

Impacted Files	Coverage Δ
coordinator/coordinator.go	42.67% <0.00%> (-0.13%)	⬇️
helpers/compression/compression.go	0.00% <0.00%> (ø)
provider/k8s.go	9.30% <ø> (-20.90%)	⬇️
provider/provider_config/spark_config.go	60.60% <ø> (+3.55%)	⬆️
provider/provider_type/provider_type.go	100.00% <ø> (ø)
provider/redshift.go	85.24% <ø> (ø)
provider/spark.go	23.25% <ø> (+4.67%)	⬆️
provider/sql.go	65.47% <ø> (-0.31%)	⬇️
runner/worker/worker.go	78.57% <ø> (ø)
serving/serialize.go	74.81% <ø> (-19.58%)	⬇️
... and 7 more

... and 15 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more