Convert to Eleventy. #836

davidlehn · 2024-03-19T03:27:29Z

Status:

Second try at converting to a static site generator.
- First try was to Jekyll in 2017: Jekyll-based json-ld.org #461.
- Much of the discussion there applies here after many years.
There are many many many SSG choices. This conversion is using Eleventy. If the community wants to use another system, it's likely these conversions are a good start with similar patterns.
What's happening in the initial commits here:
- Handles pass through of all useful (and more) files that were served before.
- Converts the runtime php indexing code to build time js code.
- Some tweaks for uniformity.
The plain apache+php hosting we use now could probably host this output with a few operational changes.
Templates and various upgrades and improvements are not yet done. Solving issues one step at a time.

Issues:

Basically same ones as in the jekyll issue.
How to handle minutes/. It's gigantic and inappropriate to mix into a static site directly.
- It is being updated on the main site AND on gh pages at https://json-ld.github.io/minutes/, but the logs note that 1.8G is beyond the 1G max and deploy might fail and it can take 20min+ to deploy. Another solution would be better. Also the "pre-wg" link is easy to not notice.
How to handle the playground and playground php proxies.
How best to host and handle updates.
- May need content type and cors fixes depending on hosting platform.

Notes:

One approach for many issues is a subdomain using gh pages or something else for minutes, cloudflare for hosting, cloudflare workers for the proxy. I think Benjamin likes this idea.
We also need to decide how to improve the playground. No one at all wants to work on it as is. Integrating a web app build step into the static site process is doable but perhaps not the best approach. I'm pondering the site code here being more content based and (ab)using a CDN or gh pages to host built playground app bundles and resources. Suggestions welcome on that.

Testing:

npm install
npm run serve
# open http://localhost:8080/ (or whatever URL it gives you)

- Copy and ignore files as needed. - Comment out old `prettify.css` usage. - Move `learn.html` to `learn/index.html` and update links. This could break external links, but is aligned with the pattern of other files and dirs. Eleventy will put it in `learn/` by default either way without an override. - `specs/`, `primer/`, and `requirements/`: - Generate sub-dir listing data with JS. - Switch from PHP to LiquidJS.

gkellogg

Thanks for work on this, Dave! Most of the space the minutes consume is due to audio captures of the early meetings, which we haven’t kept going in some time. Do logs indicate if they’re ever actually accessed? Maybe there’s a way to move those someplace else off of the main (or minutes) site.

BigBlueHat

At last!

BigBlueHat

Looks great overall. Just some non-blocking recommendations.

BigBlueHat · 2024-04-17T18:35:30Z

.eleventy.js

+  eleventyConfig.ignores.add('presentations');
+  eleventyConfig.ignores.add('scripts');
+  eleventyConfig.ignores.add('spec/tools');
+  eleventyConfig.ignores.add('spec/LICENSE.md');


These ignores can be moved to .eleventyignore to make this file more sane.

What's wrong with having them here? At least for now. It's a bit confusing as is due to eleventy processing various things, having to pass through others, and ignore particular files. And for drafts, that's done computationally. I thought it made more sense to keep the details in one place.

I get the single place to look. I mostly just like data in inert places whenever possible vs. buried among API calls.

package.json

cloudflare-workers-and-pages · 2024-04-17T18:43:14Z

Deploying json-ld-org with Cloudflare Pages

Latest commit:	`a66b568`
Status:	✅ Deploy successful!
Preview URL:	https://99ef1971.json-ld-org.pages.dev

View logs

- Convert README from reStructuredText to Markdown. - The reStructuredText format still works, and was a fine choice back in 2010(!) but Markdown is more common for READMEs like this today.

- Add `404.html` for Cloudflare Pages.

davidlehn · 2024-05-09T16:52:24Z

Another issue with this conversion is how to handle all the .htaccess rules. If the target host is Apache then of course these will still work. But our likely new target is Cloudflare Pages, which does things differently.

$ git ls-files | grep .htaccess
.htaccess
contexts/.htaccess
images/.htaccess
spec/ED/.htaccess
spec/latest/.htaccess
test-suite/.htaccess
test-suite/tests/.htaccess
utils/.htaccess

CF has _redirects and _headers files, which are simpler and limited. Some of the .htaccess rules can be handled with those files. (Untested, but looks promising)
The content type detection is trickier and I believe would require another _functions worker for some special cases. Effectively writing some js to do what apache rules do.

And while pondering these things, I'm wondering where this is headed.

A good dev env is now rather complex.
11ty has the nice auto update dev server. But it's unaware of these htaccess or cloudflare rules. Making it difficult to actually test the entire site. To use the playground proxy (see other PR), you need to have a 11th proxy to a cloudflare wrangler dev server.
Cloudflare wrangler could also serve the static dev site with its redirects and headers and functions. But then the auto update feature doesn't work. There may be some extra proxying to make that work. But then that would require two dev servers to do dev work, which seems a bit awkward.
Should we make apache or cloudflare a build option? I suspect it will just flip which files get copied to the output. But that's two things to maintain and keep in sync.

BigBlueHat · 2024-07-29T15:12:04Z

@davidlehn per your concerns above, I think we can keep using 11ty separate from the hosting platform. Having 11ty in place makes working on (and redesigning) the site much simpler--where it's hosted can be a separate matter because 11ty generates static sites.

Cloudflare hosting can easily get more complex/tangled (just like the build tool nightmare that is modern JavaScript vs. ye olde <script> tag). It really depends on what we want to manage, pay for, support, etc. Most/all of the Cloudflare stuff, though, can also be defined descriptively (or at least via JS in the same repo) to avoid "blind"/hidden configurations available only via the Cloudflare Dashboard.

All that to say, how this is hosted is orthogonal to using 11ty.

README.md

BigBlueHat · 2024-09-24T22:46:23Z

@davidlehn how close do you think this is to shippable? I know #840 is part of the consideration here too. Just curious how we can get this over the finish line.

Since the WG is getting rechartered, it'd be super to get this in a space we can continue to improve, redesign, etc.

Happy to help finish it off...just not sure where we're stuck.

- Remove PHP proxy code. - Move proxy URL to /playground/proxy. - Use eleventy@3 alpha for `onRequset` dev server support. - Add Eleventy dev server `onRequest` proxy to local wrangler server proxy. Due to current limited API, emulate a basic playground request. - Reimplement proxy as a Cloudflare Pages Function. - Basic proxy designed for limited use and abuse. - Add various checks to limit useful scope. - Update README. - Ignore .wrangler.

BigBlueHat · 2025-04-02T15:24:12Z

11ty has the nice auto update dev server. But it's unaware of these htaccess or cloudflare rules. Making it difficult to actually test the entire site. To use the playground proxy (see other PR), you need to have a 11th proxy to a cloudflare wrangler dev server.

Cloudflare wrangler could also serve the static dev site with its redirects and headers and functions. But then the auto update feature doesn't work. There may be some extra proxying to make that work. But then that would require two dev servers to do dev work, which seems a bit awkward.

To untangle this, I've added a npm run pages option which can be used alongside npm run watch. That should provide the handy auto-building from 11ty while serving a localhost Cloudflare-like environment.

Should we make apache or cloudflare a build option? I suspect it will just flip which files get copied to the output. But that's two things to maintain and keep in sync.

I don't think we should do this as it'd be a good deal of overhead to maintain/juggle both. Consequently, I've removed the .htaccess files in #855.

Once #855 is merged into this PR, I think this one can be marked "ready for review", and we can get a preview URL on Cloudflare sporting all the things for further testing from everyone here.

TallTed

My earlier concerns were due to misinterpreting Markdown. They're resolved

This uses Cloudflare `wrangler` to serve the `_site/` folder with support for `_headers` and `_redirects` files.

Cloudflare already supports the `.jsonld` extension properly (along with any/all other registered file exstensions). Cloudflare also turns CORS on by defualt for all origins.

The `images/README.md` was already handled, so the only addition to the ignores list is the `images/Makefile`.

This one never did what it intended to, judging by the commit it was added during: 5828deb

Some handled slashes, some did not. These should match what was previously in place.

CSS and images were not loading previously.

Sadly, Cloudflare currently only does content negotiation for images. However, the Pages Functions Middleware option is more than sufficient to handle the mapping and loading based on simple Accept header values. NOTE: the current implementation *is not* robust and only handles single value Accept headers.

This commit uses code generated by a Google AI Overview result. They cite no sources and claim no license. However, it is exactly what I was looking for with that search.

Not sure we need to keep this around...but I suppose it could be handy all the same.

BigBlueHat · 2025-04-14T14:12:24Z

@davidlehn I think I've addressed most if not all of your concerns in #836 (comment) via #855, so I've marked this as ready for review.

Please let me know what else it needs. Thanks!

Functional preview available at https://99ef1971.json-ld-org.pages.dev/

BigBlueHat · 2025-04-14T14:36:44Z

Clicking back through the site, I realized we never answered this one:

How to handle minutes/. It's gigantic and inappropriate to mix into a static site directly.

It is being updated on the main site AND on gh pages at https://json-ld.github.io/minutes/, but the logs note that 1.8G is beyond the 1G max and deploy might fail and it can take 20min+ to deploy. Another solution would be better. Also the "pre-wg" link is easy to not notice.

https://99ef1971.json-ld-org.pages.dev/minutes/ (in the preview) is current 404.

The simplest solution would be to setup a redirect from that URL to https://json-ld.github.io/minutes/ (where it's actually hosted).

Alternatively, if we think the json-ld.org URLs are important, we could proxy those URLs through a Cloudflare Pages Function (though that likely comes with some bandwidth costs)--which should not incur additional costs (afaict from https://developers.cloudflare.com/workers/platform/pricing/).

Or, similarly, we could push the files to a Cloudflare R2 public bucket (which supports large static file hosting similar to S3). However, that may eventually incur some cost or at minimum we'd have to be strategic in how we setup publishing the files to R2--so we don't ship all 1.8 GB every time a new change is made to the minutes...

My current vote would be a simple redirect setup to keep thing uncomplicated.

Thoughts @davidlehn @gkellogg?

BigBlueHat · 2025-04-23T18:21:52Z

It's time.

davidlehn added 2 commits March 18, 2024 21:49

Remove empty file.

19883af

davidlehn added the website label Mar 19, 2024

gkellogg approved these changes Mar 19, 2024

View reviewed changes

davidlehn marked this pull request as draft March 19, 2024 03:50

BigBlueHat approved these changes Apr 17, 2024

View reviewed changes

davidlehn added 4 commits April 25, 2024 21:03

Use installed eleventy.

ba19a52

Remove unused fields.

386eb14

Convert README to markdown.

b0fa2a8

- Convert README from reStructuredText to Markdown. - The reStructuredText format still works, and was a fine choice back in 2010(!) but Markdown is more common for READMEs like this today.

Add 404 page.

a8a8b0c

- Add `404.html` for Cloudflare Pages.

TallTed requested changes Jul 29, 2024

View reviewed changes

README.md Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

davidlehn added 2 commits March 27, 2025 11:54

Ignore .backup files.

d366038

BigBlueHat requested a review from TallTed April 1, 2025 16:07

TallTed approved these changes Apr 3, 2025

View reviewed changes

BigBlueHat added 9 commits April 14, 2025 10:10

Add wrangler; upgrade eleventy to v3.0.0.

b3aa460

Ignore package-lock.json.

0dfc72b

Add .editorconfig.

2557625

Add npm run pages command.

0f90edc

This uses Cloudflare `wrangler` to serve the `_site/` folder with support for `_headers` and `_redirects` files.

Add info about npm run build & pages.

0504c10

Copy over _headers and _redirects.

4162056

Migrate root .htaccess to _redirects.

7fd7d12

Cloudflare already supports the `.jsonld` extension properly (along with any/all other registered file exstensions). Cloudflare also turns CORS on by defualt for all origins.

Remove images/.htaccess; handled by ignores.

ae855ea

The `images/README.md` was already handled, so the only addition to the ignores list is the `images/Makefile`.

Migrate context/.htaccess to _headers/_redirects.

55bea6f

BigBlueHat added 16 commits April 14, 2025 10:10

Remove broken spec/ED/.htaccess.

612b36d

This one never did what it intended to, judging by the commit it was added during: 5828deb

Migrate spec/latest/.htaccess.

39d6139

Some handled slashes, some did not. These should match what was previously in place.

Absolutize URLs in 404 page.

f40f137

CSS and images were not loading previously.

Parse and sort accept headers; pick best q.

df0ce8e

This commit uses code generated by a Google AI Overview result. They cite no sources and claim no license. However, it is exactly what I was looking for with that search.

Migrate test-suite/tests/ redirs & headers.

731c01f

Remove utils/; No longer needed.

b91a537

Switch to type: module.

3e7ac45

Add initial test-redirects.js script.

515c98a

Not sure we need to keep this around...but I suppose it could be handy all the same.

Test Link headers via _headers.

717bd90

Optimize _headers order and format.

545d6dc

Reorder test-redirects.js; group test types.

1297213

Test all redirects in _redirects.

b4ebf27

Optimize redirect order; https for schema.org.

4fedf47

Set wrangler compatibility date.

6934340

Remove eleventy --serve; promote using run pages.

a66b568

BigBlueHat marked this pull request as ready for review April 14, 2025 14:11

BigBlueHat merged commit d4b493a into main Apr 23, 2025
1 check passed

BigBlueHat deleted the 11ty branch April 23, 2025 18:21

This was referenced Apr 23, 2025

Add redir from /minutes/ to minutes subdomain. #859

Merged

Jekyll-based json-ld.org #461

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert to Eleventy. #836

Convert to Eleventy. #836

davidlehn commented Mar 19, 2024

gkellogg left a comment

BigBlueHat left a comment

BigBlueHat left a comment

BigBlueHat Apr 17, 2024

davidlehn Apr 26, 2024

BigBlueHat Jul 29, 2024

cloudflare-workers-and-pages bot commented Apr 17, 2024 •

edited

Loading

davidlehn commented May 9, 2024

BigBlueHat commented Jul 29, 2024

BigBlueHat commented Sep 24, 2024

BigBlueHat commented Apr 2, 2025

TallTed left a comment

BigBlueHat commented Apr 14, 2025 •

edited

Loading

BigBlueHat commented Apr 14, 2025

BigBlueHat commented Apr 23, 2025

Convert to Eleventy. #836

Convert to Eleventy. #836

Conversation

davidlehn commented Mar 19, 2024

gkellogg left a comment

Choose a reason for hiding this comment

BigBlueHat left a comment

Choose a reason for hiding this comment

BigBlueHat left a comment

Choose a reason for hiding this comment

BigBlueHat Apr 17, 2024

Choose a reason for hiding this comment

davidlehn Apr 26, 2024

Choose a reason for hiding this comment

BigBlueHat Jul 29, 2024

Choose a reason for hiding this comment

cloudflare-workers-and-pages bot commented Apr 17, 2024 • edited Loading

Deploying json-ld-org with Cloudflare Pages

davidlehn commented May 9, 2024

BigBlueHat commented Jul 29, 2024

BigBlueHat commented Sep 24, 2024

BigBlueHat commented Apr 2, 2025

TallTed left a comment

Choose a reason for hiding this comment

BigBlueHat commented Apr 14, 2025 • edited Loading

BigBlueHat commented Apr 14, 2025

BigBlueHat commented Apr 23, 2025

cloudflare-workers-and-pages bot commented Apr 17, 2024 •

edited

Loading

BigBlueHat commented Apr 14, 2025 •

edited

Loading