overhaul build system

[Kriechi]

The current build system is using snowpack, which is deprecated and has not received updates in the last two years.
Currently, the repository contains a compiled/generated copy of all resources which is outdated and can cause version mismatches due to race conditions during compilation with snowpack (see FredKSchott/snowpack#3289). This introduces a non-deterministic deployment to be pushed out, potentially containing old (insecure) code. Furthermore, developing locally causes a constant diff against these files.

This PR removes the stale copy of compiled resources from the repository.

This PR migrates from the deprecated snowpack to the actively maintained vite build system, as recommended by snowpack themselves as possible upgrade path (see https://github.com/FredKSchott/snowpack/blob/main/README.md).

This PR restructures the assets folders to be top-level for each resource type, which fits better with vite.

This PR brings in all 3rd party dependencies and libraries to be part of the compiled single page app. No more online downloads from CDNs.

This PR also updates the necessary GitHub Actions workflow files - these need additional testing which I cannot run as part of this PR alone.

[makermelissa]

Looks like this isn't passing the checks because it can't find the files. If you can get this working and the site functions without errors, I'm good with these changes.

[Kriechi]

This is a super annoying bug in snowpack:
FredKSchott/snowpack#3289

I thought I fixed it, as it worked locally, but it seems the timing is just different enough in GitHub Actions to be broken again. I actually believe is is generally broken on the main branch as well, as there is no guarantee that the build+deploy is actually from the expected version of files, but it might as well just use the old version still in the repo. Not deterministic in my understanding.

I spent hours trying to fix this today - but at this point I'm wondering if a different tool chain would be more appropriate?

Update (April 20, 2022): Snowpack is no longer actively maintained and is not recommended for new projects.

[makermelissa]

@tannewt, since you originally chose snowpack, what are your thoughts?

[Kriechi]

@makermelissa @tannewt I updated the PR and migrated from snowpack to vite. Please read through the main PR description for all new changes in this PR.

[makermelissa]

I did a little more looking into this yesterday and I think this was a good choice of build system to go with. I would just like to get Scott's thoughts first, and I'm not sure his ETA for being back. Also, I'm going to be out most of this next week, so it may not be a good idea to merge right before that in the case of something unexpectedly breaking that needs fixing right away. So because this is a major change and the above reasons, I may hold out on merging until next week.

[tannewt]

I don't have any affinity for snowpack or experience with JS. So I'd leave it up to you both.

[makermelissa]

Ok cool. Works for me. There's a few merge conflicts that need to be fixed first, then I'd like to test this locally, and then I think this will be good to go.

[Kriechi]

rebased & fixed conflicts.

Please test extensively, I only know and did the basics using USB, but I did not test the other workflows at all or in depth.
Would be nice to have some automated tests in the future!

[Kriechi]

updated and rebased to recent main branch.

[thess]

I don't think this works with the current web-workflow. CP devices directly load device.js via the embedded code.html page.
https://github.com/adafruit/circuitpython/blob/main/supervisor/shared/web_workflow/static/code.html

There must be a way to expose this script using vite (of which I know almost nothing).
Also, you might want to make smaller images to avoid the build warning:
(!) Some chunks are larger than 500 kBs after minification.

Adding this stanza (found on the 'net) seems to work ok.

    build: {
        rollupOptions: {
            output:{
                manualChunks(id) {
                    if (id.includes('node_modules')) {
                        return id.toString().split('node_modules/')[1].split('/')[0].toString();
                    }
                }
            }
        }
    }

That said, I did successfully build and deploy your changes and web-workflow works if invoked directly but not via the CP device landing page (404 error on device.js)

[makermelissa]

Good catch @thess. That one isn't very obvious.

[Kriechi]

@thess would this generated folder structure work for the web workflow?

dist/
├── assets/
│   ├── fa-brands-400-20c4a58b.ttf
│   ├── fa-brands-400-74833209.woff2
│   ├── fa-regular-400-528d022d.ttf
│   ├── fa-regular-400-8e7e5ea1.woff2
│   ├── fa-solid-900-67a65763.ttf
│   ├── fa-solid-900-7152a693.woff2
│   ├── fa-v4compatibility-0515a423.ttf
│   ├── fa-v4compatibility-694a17c3.woff2
│   ├── favicon-badb8633.ico
│   ├── index-1ac6c131.css
│   ├── index-b1ab6ff1.js
│   ├── js/
│   │   └── device.js
│   ├── loading-blinka-0bd52a91.gif
│   ├── loading-blinka-d0565150.webp
│   ├── logo@2x-78412641.png
│   └── logo@3x-c5751f91.png
└── index.html

[makermelissa]

I believe so. The device.js file must be accessible at https://code.circuitpython.org/assets/js/device.js. The purpose of this script is to bootstrap load the website via javascript as if it is loading it right on the device. You can test by loading the site locally as secure and going to https://localhost:8080/code/#host=circuitpython.local with a device plugged in. Change circuitpython.local as needed for the device host name.

[makermelissa]

Actually, it looks like I only have the code/index.html on my computer locally. Here's the contents:

<!doctype html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <link rel="icon" href="https://code.circuitpython.org/assets/images/favicon.ico" type="image/x-icon" />
        <title>Online Code Editor</title>
    </head>

    <body>
        <script type="module" src="http://localhost:8080/assets/js/device.js"></script>
        <p>Uh oh! It looks like you may be offline. You can go to the <a href="/fs/">file browser</a> if you'd like to work with files without connecting to the internet.</p>
    </body>
</html>

[Kriechi]

Thanks - then this should work just fine. I already updated the PR to generate exactly the folder structure as shown above. I simple moved the device.js file into the special-purpose public/ folder for vite to copy it as-is into the output dist/ folder with the correct intermediary sub-folder structure.

[thess]

@Kriechi - Looks good, thanks for this
@makermelissa - thanks for the debug hints.

[thess]

One more item -- favicon.ico needs to be exposed in assets/images/favicon.ico for the same reason as device.js was (referenced by static/embeddded code.html)

[Kriechi]

Done!

[thess]

Sorry for being such a nag -- I'm not very well versed in this environment.

I noticed the logo is not rendering in the vite production build when served by a regular web server like nginx. The embedded base64 PNG is causing 404 errors. Moving the logo??.png files to the public folder and fixing up index.html to find them seems to solve this issue.

I hope this is the last tweak on this PR.

[Kriechi]

Fixed!

New cleaned up folder structure:

dist
├── assets
│   ├── fonts
│   │   ├── fa-brands-400.ttf
│   │   ├── fa-brands-400.woff2
│   │   ├── fa-regular-400.ttf
│   │   ├── fa-regular-400.woff2
│   │   ├── fa-solid-900.ttf
│   │   ├── fa-solid-900.woff2
│   │   ├── fa-v4compatibility.ttf
│   │   └── fa-v4compatibility.woff2
│   ├── images
│   │   ├── favicon.ico
│   │   ├── loading-blinka.gif
│   │   ├── loading-blinka.webp
│   │   ├── logo.png
│   │   ├── logo@2x.png
│   │   └── logo@3x.png
│   ├── index.css
│   └── js
│       ├── device.js
│       └── index.js
└── index.html

[makermelissa]

This looks really good. The only change that I would like to see is an option to be able to run this as https by passing a command. This was something I used frequently prior to this PR and this PR appears to remove that functionality.

As it is, chrome doesn't accept the SSL cert just by passing --https as an option to vite. An alternate config that allows use of generated self signs certificates would probably be easiest.

[Kriechi]

Curious what use cases you have for HTTPS specifically?
At least the specific web-serial APIs are fully functional when using "localhost".

Without having used or reviewed them myself, these seem to be popular plugins for vite:
https://github.com/vitejs/vite-plugin-basic-ssl
and
https://github.com/liuweiGL/vite-plugin-mkcert

If one of these fits your needs and previous workflow, I can link them in the README (again).

[makermelissa]

Mostly because the final site is https://code.circuitpython.org and it's useful to test because sometimes CORS issues come up where they may not on http or vice versa. It's also sometimes useful to test it as http, which is why I'd like the option.

The mkcert option sounds good. Thanks.

[Kriechi]

@makermelissa you can probably keep that in a local (not committed to the repo) config file, and pass it to vite using a command like npx vite --config secure-config.js dev
https://vitejs.dev/config/

Or do you want to make it part of the default config? Not sure how many actually need it.

[makermelissa]

@makermelissa you can probably keep that in a local (not committed to the repo) config file, and pass it to vite using a command like npx vite --config secure-config.js dev https://vitejs.dev/config/

Or do you want to make it part of the default config? Not sure how many actually need it.

This sounds like a good solution.

[makermelissa]

Please fix the merge conflicts and then I'm happy to merge this.

[Kriechi]

rebased & no more conflicts!

[makermelissa]

Thank you for your hard work on this.