feat: serve AsyncAPI JSON Schema definitions #680

smoya · 2022-04-22T15:53:02Z

Description

This PR let's our website to serve all AsyncAPI JSON Schema definition files from it.

All AsyncAPI JSON Schema definition files are being served within the /definitions/<file> path. The content is literally being served from GH, in particular from https://github.com/asyncapi/spec-json-schemas/tree/master/schemas.
This is possible thanks to the following:

A Netlify Rewrite rule located in the netlify.toml file, which acts as proxy for all requests to the /definitions/<file> path, serving the content from GH without having an HTTP redirect.
A Netlify Edge Function that modifies the Content-Type header of the rewrite response to become application/schema+json. This lets tooling, such as Hyperjump, to fetch the schemas directly from their URL.

In order to see what the final result is, just grab the Netlify build preview link and run:

curl -v <preview-url>/definitions/2.3.0.json

Btw, this Netlify edge function could be used also for pushing metrics on the hit count, solving #780.

Related issue(s)
Iteration over #502

cc @jonaslagoni @derberg @fmvilas @magicmatatjahu

netlify · 2022-04-22T15:53:09Z

✅ Deploy Preview for asyncapi-website ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`e76f8f6`
🔍 Latest deploy log	https://app.netlify.com/sites/asyncapi-website/deploys/62ab034265a33300082a4d3f
😎 Deploy Preview	https://deploy-preview-680--asyncapi-website.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

github-actions · 2022-04-22T15:59:52Z

⚡️ Lighthouse report for the changes in this PR:

Category	Score
🔴 Performance	32
🟢 Accessibility	98
🟠 Best practices	83
🟢 SEO	100
🔴 PWA	30

Lighthouse ran on https://deploy-preview-680--asyncapi-website.netlify.app/

README.md

quetzalliwrites

just one tiny suggestion @smoya 🐼

derberg · 2022-04-28T09:46:30Z

Interesting approach 🤔

I understand that we can have an edge function that on each request can do anything, like API call to some service, right?
You do know it is experimental? Also have a look at -> https://github.community/t/raw-githubusercontent-com-rate-limit/142444. For me, it is actually ok because I have a totally opposite view on the High Availability argument you made here. Just want to make sure you are aware

Co-authored-by: Alejandra Quetzalli <alejandra.quetzalli@postman.com>

smoya · 2022-05-02T17:15:24Z

I understand that we can have an edge function that on each request can do anything, like API call to some service, right?

Yes, that's right.

You do know it is experimental?

Yes I know, but right now it is a public beta which has been passed through the private beta stage.

For me, it is actually ok because I have a totally opposite view on the High Availability argument you made #780.

I still believe Netlify availability is pretty reliable and superior to what we can support ourselves with the people who are involved. Including using beta features such as Edge Functions.

Also have a look at -> https://github.community/t/raw-githubusercontent-com-rate-limit/142444

You made a good point here. The authentication token is going to be needed to have those 5k req/hour.
I could not make proper testing around how github counts a cached VS non-cached request in terms of rate limit because GH API /rate_limit endpoint doesn't shows such stats for raw.githubusercontent.

However, there is something to consider here and it's the fact Netlify should cache each request for "some time":

If the API supports standard HTTP caching mechanisms like ETags or Last-Modified headers, the responses will even get cached by our CDN nodes.
Source: https://docs.netlify.com/routing/redirects/rewrites-proxies/#proxy-to-another-service

If we run the following:

curl -I -H "Authorization: $MY_TOKEN" https://raw.githubusercontent.com/asyncapi/spec/v2.4.0/examples/websocket-gemini.yml

HTTP/2 200
cache-control: max-age=300
content-security-policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
content-type: text/plain; charset=utf-8
etag: "755fbfddca88cfd66072e57e21f9a33f3fabbe6ae24ce7c0249d83ab8d3d7788"
strict-transport-security: max-age=31536000
x-content-type-options: nosniff
x-frame-options: deny
x-xss-protection: 1; mode=block
x-github-request-id: F288:9F7F:6918C6:6DD3EC:626FDD1E
accept-ranges: bytes
date: Mon, 02 May 2022 13:31:10 GMT
via: 1.1 varnish
x-served-by: cache-mad22067-MAD
x-cache: MISS
x-cache-hits: 0
x-timer: S1651498270.022946,VS0,VE205
vary: Authorization,Accept-Encoding,Origin
access-control-allow-origin: *
x-fastly-request-id: 5327fa939e275e7ddbd14bc99a8fb310421282a5
expires: Mon, 02 May 2022 13:36:10 GMT
source-age: 0
content-length: 9100

We can see the content should be cached for 5 mins (300 seconds): cache-control: max-age=300. And considering that Vary header value is Authorization,Accept-Encoding,Origin, we can also do the following:

Remove the Origin request header, so GH always serves the same cached content, no matter the Origin header.
Regarding the Accept-Encoding, we should not modify it. It is the expected by the client encoding.

That would give us a really big number of requests allowed (cached or non cached), all pending to be tested and confirmed. In the really worst scenario, we could switch to what @jonaslagoni suggested in #502 so we serve the files right from Netlify instead of GH, but keeping the Edge Function in there for the Headers change + metrics collection.

smoya · 2022-05-02T17:49:55Z

netlify/edge-functions/serve-definitions.ts

+  request.headers.delete("origin");
+
+  // Setting GH Token to increase GH rate limit to 5,000 req/h.
+  request.headers.set("Authorization", "token " + Deno.env.get("GITHUB_TOKEN"));


We would need to use a GH token here. Preferably, a token from a user is not being used in any automation yet. cc @derberg

a token from a user

what do you mean? why user token. We can use AsyncAPI token

Oh, i meant using a new user because requests to GH API are counted across the whole account, no matters wich toke you use. Important for the rate limit.

smoya · 2022-05-04T15:16:06Z

I added the following:

New /schema-store path to be used only by schemas hosted at SchemaStore.org.
Send metrics to NewRelic (More info at Measuring AsyncAPI spec adoption #780).

Some env vars will be needed to be set at Netlify level:

GITHUB_TOKEN: Preferably, a token from a user is not being used in any automation yet.
NR_ACCOUNT: the New Relic account id. Can be found on New Relic's website, path: /admin-portal/api-keys.
NR_API_KEY: The New Relic API KEY: A License type API Key that can be created on New Relic's website, path: /admin-portal/api-keys

As a side note, the URL to NewRelic insights collector should be changed in case AsyncAPI New Relic Account is not hosted in EU to https://insights-collector.newrelic.com/v1/accounts/YOUR_ACCOUNT_ID/events.

Here a screenshot of how our dashboard could look in New Relic:

derberg · 2022-05-12T10:20:58Z

I think we need some page under https://deploy-preview-680--asyncapi-website.netlify.app/schema-store that explains the concept and lists all available schemas

also to go further lets make sure others have nothing against new relic and hosting from GitHub Raw atm -> #780

smoya · 2022-06-10T10:03:03Z

I think we need some page under https://deploy-preview-680--asyncapi-website.netlify.app/schema-store that explains the concept and lists all available schemas

Who could help on that part?

also to go further lets make sure others have nothing against new relic and hosting from GitHub Raw atm -> #780

It seems people agreed on this finally!

smoya · 2022-06-14T16:01:18Z

@derberg redirects have been added to the schemas README from asyncapi/spec-json-schemas#229, which will be located at https://github.com/asyncapi/spec-json-schemas/tree/master/schemas

smoya · 2022-06-14T16:28:10Z

This is ready for review. Tested locally through netlify dev.

netlify/edge-functions/serve-definitions.ts

Co-authored-by: Lukasz Gornicki <lpgornicki@gmail.com>

smoya · 2022-06-15T17:59:31Z

@derberg I checked the retention New Relic provides to the free and standard tier:

Source: https://newrelic.com/pricing

As I was sending custom events, I decided to change it to send metrics instead, so we can take profit of the 13 months of retention. That means the code changed a bit so you will need to review it again (sorry for that).

Also, the NR_ACCOUNT env var is no longer needed with this new approach.

fmvilas

This is really cool 🔝

Everything looks good to me. Do you want to wait for @derberg's review?

derberg

❤️
Once it is merged we need to open PR to schemastore

smoya · 2022-06-15T18:23:52Z

❤️ Once it is merged we need to open PR to schemastore

I'm happy to do it

smoya · 2022-06-15T18:53:06Z

/rtm

smoya · 2022-06-15T19:07:17Z

/rtm

smoya · 2022-06-15T19:07:52Z

/help

asyncapi-bot · 2022-06-15T19:08:07Z

Hello, @smoya! 👋🏼

I'm Genie from the magic lamp. Looks like somebody needs a hand! 🆘

At the moment the following comments are supported in pull requests:

/ready-to-merge or /rtm - This comment will trigger automerge of PR in case all required checks are green, approvals in place and do-not-merge label is not added
/do-not-merge or /dnm - This comment will block automerging even if all conditions are met and ready-to-merge label is added
/autoupdate or /au - This comment will add autoupdate label to the PR and keeps your PR up-to-date to the target branch's future changes. Unless there is a merge conflict or it is a draft PR.

smoya · 2022-06-15T19:14:54Z

@alequetzalli I need your review again after all suggestions were applied. 🙏 thank you!

smoya · 2022-06-16T11:52:57Z

❤️ Once it is merged we need to open PR to schemastore

Changes are pushed to my fork but waiting for this PR to be merged (approval by @alequetzalli is required) to be turned into a PR. https://github.com/SchemaStore/schemastore/compare/master...smoya:feat/asyncapiSchemaStore?expand=1

quetzalliwrites

🤪🥳🫡

smoya · 2022-06-21T07:45:12Z

/rtm

smoya requested review from quetzalliwrites, asyncapi-bot-eve, fmvilas, derberg and magicmatatjahu as code owners April 22, 2022 15:53

smoya mentioned this pull request Apr 22, 2022

Measuring AsyncAPI spec adoption #780

Open

quetzalliwrites reviewed Apr 22, 2022

View reviewed changes

README.md Outdated Show resolved Hide resolved

quetzalliwrites reviewed Apr 22, 2022

View reviewed changes

smoya and others added 3 commits May 2, 2022 15:36

feat: serve AsyncAPI JSON Schema definitions

86821a2

fix wrong dir

0dbfa26

Update README.md

25c1e03

Co-authored-by: Alejandra Quetzalli <alejandra.quetzalli@postman.com>

smoya force-pushed the feat/definitions branch from a697a42 to 25c1e03 Compare May 2, 2022 13:36

authorization + cleanup headers

b8c463b

smoya commented May 2, 2022

View reviewed changes

smoya added 5 commits May 4, 2022 16:09

Merge branch 'master' into feat/definitions

b2eccda

add a new path /schemas for definitions fetched from schemastore.org

9e994c2

send metrics to NR

4ce68cb

rename /schemas to /schema-store

202ae0d

extract event generation

157636d

jonaslagoni mentioned this pull request May 17, 2022

Work on 3.0 release asyncapi/spec#691

Closed

Merge branch 'master' into feat/definitions

4cd35fd

quetzalliwrites previously approved these changes Jun 10, 2022

View reviewed changes

Merge branch 'master' into feat/definitions

e2c1287

notify NR on errors

6dd2945

smoya dismissed quetzalliwrites’s stale review via 6dd2945 June 13, 2022 15:20

redirect to README in / path

5c46144

add url field to NR event

9df31a4

Merge branch 'master' into feat/definitions

a3ffda9

derberg reviewed Jun 15, 2022

View reviewed changes

netlify/edge-functions/serve-definitions.ts Outdated Show resolved Hide resolved

smoya and others added 2 commits June 15, 2022 19:28

Update netlify/edge-functions/serve-definitions.ts

dd20b7a

Co-authored-by: Lukasz Gornicki <lpgornicki@gmail.com>

Use New Relic Metrics API Instead of events (retention is way higher)

a683064

lint

2f92295

fmvilas approved these changes Jun 15, 2022

View reviewed changes

derberg approved these changes Jun 15, 2022

View reviewed changes

asyncapi-bot added the ready-to-merge label Jun 15, 2022

smoya requested a review from quetzalliwrites June 15, 2022 19:14

Merge branch 'master' into feat/definitions

e76f8f6

asyncapi-bot removed the ready-to-merge label Jun 16, 2022

quetzalliwrites approved these changes Jun 17, 2022

View reviewed changes

asyncapi-bot added the ready-to-merge label Jun 21, 2022

asyncapi-bot merged commit 60a698b into asyncapi:master Jun 21, 2022

smoya deleted the feat/definitions branch June 21, 2022 07:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: serve AsyncAPI JSON Schema definitions #680

feat: serve AsyncAPI JSON Schema definitions #680

smoya commented Apr 22, 2022

netlify bot commented Apr 22, 2022 •

edited

Loading

github-actions bot commented Apr 22, 2022 •

edited

Loading

quetzalliwrites left a comment

derberg commented Apr 28, 2022

smoya commented May 2, 2022 •

edited

Loading

smoya May 2, 2022

derberg May 12, 2022

smoya May 12, 2022

smoya commented May 4, 2022

derberg commented May 12, 2022

smoya commented Jun 10, 2022 •

edited

Loading

smoya commented Jun 14, 2022 •

edited

Loading

smoya commented Jun 14, 2022

smoya commented Jun 15, 2022

fmvilas left a comment

derberg left a comment

smoya commented Jun 15, 2022

smoya commented Jun 15, 2022

smoya commented Jun 15, 2022

smoya commented Jun 15, 2022

asyncapi-bot commented Jun 15, 2022

smoya commented Jun 15, 2022

smoya commented Jun 16, 2022

quetzalliwrites left a comment

smoya commented Jun 21, 2022

feat: serve AsyncAPI JSON Schema definitions #680

feat: serve AsyncAPI JSON Schema definitions #680

Conversation

smoya commented Apr 22, 2022

netlify bot commented Apr 22, 2022 • edited Loading

✅ Deploy Preview for asyncapi-website ready!

github-actions bot commented Apr 22, 2022 • edited Loading

quetzalliwrites left a comment

Choose a reason for hiding this comment

derberg commented Apr 28, 2022

smoya commented May 2, 2022 • edited Loading

smoya May 2, 2022

Choose a reason for hiding this comment

derberg May 12, 2022

Choose a reason for hiding this comment

smoya May 12, 2022

Choose a reason for hiding this comment

smoya commented May 4, 2022

derberg commented May 12, 2022

smoya commented Jun 10, 2022 • edited Loading

smoya commented Jun 14, 2022 • edited Loading

smoya commented Jun 14, 2022

smoya commented Jun 15, 2022

fmvilas left a comment

Choose a reason for hiding this comment

derberg left a comment

Choose a reason for hiding this comment

smoya commented Jun 15, 2022

smoya commented Jun 15, 2022

smoya commented Jun 15, 2022

smoya commented Jun 15, 2022

asyncapi-bot commented Jun 15, 2022

smoya commented Jun 15, 2022

smoya commented Jun 16, 2022

quetzalliwrites left a comment

Choose a reason for hiding this comment

smoya commented Jun 21, 2022

netlify bot commented Apr 22, 2022 •

edited

Loading

github-actions bot commented Apr 22, 2022 •

edited

Loading

smoya commented May 2, 2022 •

edited

Loading

smoya commented Jun 10, 2022 •

edited

Loading

smoya commented Jun 14, 2022 •

edited

Loading