Feature request: support Open Graph Protocol

[hdoordt]

Crates.io currently doesn’t have support for the Open Graph Protocol (https://ogp.me/), meaning that when linking to a crate on social media, a default image that is not too informative gets shown. To improve this situation, support for Open Graph metadata should be added to the HTML served from crate pages.

As it turns out, crates.io’s static HTML index file is served by the backend, and introducing a simple HTML template could enable support for the Open Graph protocol relatively trivially. I’d be happy to open a PR that adds support if desired. Actually generating the referred-to images can be left to a separate service, for which I have already written a small prototype. I’ve published that prototype here.

Although performance might be negatively affected by having the backend render HTML specific to a crate page, the setup could benefit greatly from caching throughout the stack.

With some pointers from the team, I’ll gladly work on implementing this.

[Turbo87]

Crates.io currently doesn’t have support for the Open Graph Protocol

just to be pedantic, we do include <meta property="og:image" content="https://crates.io/assets/og-image.png"> in our HTML, but you are correct that it's the same for every page at the moment.

introducing a simple HTML template could enable support for the Open Graph protocol relatively trivially
[...]
Although performance might be negatively affected by having the backend render HTML specific to a crate page, the setup could benefit greatly from caching throughout the stack.

• what happens when users request a page of a crate that does not exist?
• how will the caching work?
• we've successfully used https://github.com/moka-rs/moka in the past, but that is caching per server and can't easily be invalidated across our servers. question is, do we need cache invalidation for this?

Actually generating the referred-to images can be left to a separate service

I guess it would probably be easiest for them to get generated by a background job and then pushed to S3, so that we don't have to generate them dynamically.

written a small prototype

first glance feedback: the download number has a high update frequency, but we probably wouldn't want to rerender these images very often, so I'm not sure whether we should include it in the image.

With some pointers from the team, I’ll gladly work on implementing this.

let me know what you need :)

[hdoordt]

Thanks for the reply @Turbo87!

Some reactions to your questions:

what happens when users request a page of a crate that does not exist?

Good call. I think we want to keep away from doing database access in the serve_html middleware for performance reasons. Instead, that middleware would dumbly try to parse the URL so as to figure out the requested crate name and version. We may even want to do that at the router level instead, depending on what's most performant. In case parsing fails, it could return the HTML as it is currently, unmodified, referring to the current OG image.

In case parsing succeeds, though, the og:image location would be set to https://domain.of.og-loc.example/og/:name/:version and clients will attempt to fetch the OG image from there. The OG image generator does need to determine the existence of a crate name and version combination in any case, and could thus serve a 404-specific image in case that fails, providing users with some feedback in case they made a typo for instance.

how will the caching work?

This is very much dependent on the current deployment setup, and I'll need some pointers for this. In the ideal situation, we could leverage the CDN, but pricing and performance are very important factors here. However, I'd like for it to be set up in such a way that the rendering service doesn't get invoked at all in case an image already exists. Where can I find detailed info on the current deployment setup?

we've successfully used https://github.com/moka-rs/moka in the past, but that is caching per server and can't easily be invalidated across our servers. question is, do we need cache invalidation for this?

I don't think we need explicit cache invalidation. That would only be necessary in case we end up redesigning the image, but even then it's not imperative those changes propagate immediately. So the cache TTL could be pretty long.

I guess it would probably be easiest for them to get generated by a background job and then pushed to S3, so that we don't have to generate them dynamically.

If that's easiest to set up, then yeah I'd say let's go for it. However, with the right caching setup I think rendering on demand shouldn't impose much of a bottleneck. And in case we do want to update the OG image design, there'd be no need to render all images at once.

first glance feedback: the download number has a high update frequency, but we probably wouldn't want to rerender these images very often, so I'm not sure whether we should include it in the image.

Yup, let's leave that kind of stuff out for now

[Turbo87]

The OG image generator does need to determine the existence of a crate name and version combination in any case

do we want per-version OG images or is per-crate enough?

could thus serve a 404-specific image in case that fails

if we served these images through the CDN then a fallback image logic would have to be implemented in https://github.com/rust-lang/simpleinfra/. I would assume that something like this should be possible, but this would have to be researched.

This is very much dependent on the current deployment setup, and I'll need some pointers for this

• we currently use two Heroku dynos for the API and frontend, another one for the background worker
• we are hosting our Postgres database at https://crunchybridge.com these days
• there is currently no Redis server or anything like that in place for caching across the dynos.
• we are using S3 for file storage
• CloudFront is (currently) used in front of the Heroku dynos for the API and frontend
• Fastly is (currently) primarliy used in front of S3, with a small portion of the traffic still going through CloudFront

In the ideal situation, we could leverage the CDN

for the images that should work, but I'm a bit worried about caching the generated HTML files on the CDNs, since deploying new frontend versions would then need an explicit invalidation, while we're currently just telling the CDNs to not cache the HTML and only cache the other frontend assets.

And in case we do want to update the OG image design, there'd be no need to render all images at once.

if it's implemented as a background job we could just queue up low-priority background jobs for all the crates. it doesn't necessarily have to be done all at once :)

[hdoordt]

do we want per-version OG images or is per-crate enough?

Per-version is nice if you'd like to announce a new version of your crate, but I wouldn't say it's a hard requirement. So if it turns out to be too hard to do, I think we can make do with per-crate images. But I don't think we can do without the description.

An upside of doing per-version OG images is that it allows crate owners with a way to have the image re-generated each time they update the crate, allowing them to alter the description and other info at some point. But we could also allow crate owners to regenerate crate images on demand with a rate limit.

if we served these images through the CDN then a fallback image logic would have to be implemented in https://github.com/rust-lang/simpleinfra/. I would assume that something like this should be possible, but this would have to be researched.

I'll have a look 👍

for the images that should work, but I'm a bit worried about caching the generated HTML files on the CDNs, since deploying new frontend versions would then need an explicit invalidation, while we're currently just telling the CDNs to not cache the HTML and only cache the other frontend assets.

Do you think we can get away with having the backend render the HTML for every request? In that case we'd be out out of the woods.

if it's implemented as a background job we could just queue up low-priority background jobs for all the crates. it doesn't necessarily have to be done all at once :)

Perfect!

[Turbo87]

An upside of doing per-version OG images is that it allows crate owners with a way to have the image re-generated each time they update the crate

the same could be true for per-crate images though. basically, whenever we update the default_versions table we should probably rerender the per-crate image to show the data from the crate and the current default version. (note that not all data is currently taken from the default version, some is taken from the latest published version, which IMHO is a bug. see also #953).

Do you think we can get away with having the backend render the HTML for every request? In that case we'd be out out of the woods.

if we don't want database interaction in the request handler then using something like moka for per-dyno caching of the rendered HTML should be possible. I don't know what the performance impact would be for skipping the caching and "rendering" the HTML for each request. I would assume we'd read the HTML file on startup and prepare it in a way where injecting the right image URL is "cheap", but idk yet how exactly this would look like either.

[hdoordt]

the same could be true for per-crate images though. basically, whenever we update the default_versions table we should probably rerender the per-crate image to show the data from the crate and the current default version. (note that not all data is currently taken from the default version, some is taken from the latest published version, which IMHO is a bug. see also #953).

If we're going for per-crate OG images and show the default version on them, that's going to be confusing for version-specific pages like https://crates.io/crates/knien/0.0.1. I think if we're going for per-crate images, we should leave out the version as a whole, as well as any other version-specific info. I'd be fine with that. If it does turns out to be throwing out the child with the bath water, we can always investigate moving to per-version images.

[hdoordt]

Here's what I can do right now:

• Improve design of OG image and implement it
• Design fallback image (for 404)
• Make image generator suitable for batch jobs
• Benchmark image generator a bit and fix low hanging fruit wrt performance
• Open PR on adding templating support to the crates.io backend, as well as caching using moka

I think this feature needs some experimenting, in particular to figure out whether performance impact is within reason. Actually running the image generation job, storing the images, and configuring CDN is something that is going to be hard for me.

I'm aware the crates.io team is understaffed, and as such am offering my help with any of the other tasks, but I won't be able to do them independently as it stands. With some clear instructions I do expect to be able to figure out most of it, though

[Turbo87]

Design fallback image

wouldn't that just be the current OG image? :)

[hdoordt]

I guess why not? But maybe the designer comes up with something cool ;)

[hdoordt]

@Turbo87 I opened a draft PR at #10464

[hdoordt]

I also updated og-loc so that it renders following a new design and handles long crate names, descriptions, and many crate owners correctly.

If someone from the team could give it a go, then that'd be great. Here's an (outdated) list of crates that can be used as input to the bulk subcommand.

[Turbo87]

FWIW I think we probably wouldn't want to use https://github.com/mainmatter/og-loc directly. I think we're fine with third-party dependencies for generic libraries, but if it is generating crates.io-specific images then the code should live in the crates.io repository. we probably also wouldn't want to use it in CLI mode and use it as a library instead, but I'm not 100% decided on the pros and cons for that.

[hdoordt]

All right, that makes sense. I'd be happy to adapt the thing to your preferred way of using it, and hand it over. Just need to know how you'd like to use it.

The PR #10464 to crates.io is agnostic to the exact location of the images or the way they've been rendered for exactly this reason.

[hdoordt]

@Turbo87 thanks again for touching up #10464 and merging it. With that in place, Crates.io now supports dynamic OG images. The other part is setting up a source for the OG images. You're welcome to use or adapt og-loc, but sadly I'll be unavailable to do any significant further work on that in the short term. I'd be happy to hand the repo over as well.

Can you let me know how you'd like the OG image generation and serving to work exactly? My current take is that you'd like to generate the images in bulk and then serve them from some bucket, correct? What do you think would need adapting on og-loc's side?

[Turbo87]

Can you let me know how you'd like the OG image generation and serving to work exactly?

the short version is: I don't know yet 😂

but yeah, we would most likely save the images in our S3 bucket where e.g. the rendered readme files are also stored. we would add another background job that renders and uploads the image for a crate and then trigger that job whenever a new versions is published.

regarding og-loc, using typst looks like an interesting idea, though I'm a bit afraid of adding another big number of dependencies to our project for compile time and security reasons. if this is the most reasonable way then I'm not necessarily against it, but it might be worth looking into alternatives. I would assume that we actually only use a small subset of the typst functionality?