Improve workflow for adding projects

[2color]

Background

Currently we rely on Airtable as the source of truth for projects in this repo.

The current workflow involves:

1. Reviewing the submissions on Airtable
2. Once the approval checkbox is checked the json filed in Airtable is populated with the content from all the fields
3. The JSON is manually copied into a file in this repo described here

Challenges with the current approach

• When submissions are made to Airtable, there's no easy way to ask the submitters for changes. This puts the burden of editing on us rather than being able to provide feedback and make changes in a single place.
• Making changes requires updating Airtable, having the JSON regenerated and copying over the changes here which is also error prone
• It's not possible to preview how the website looks with new submissions in Airtable
• There's no automated way to check if all the checked projects in Airtable are published in the ecosystem website
• Moving the JSON from Airtable is error prone and requires opening a PR in this repository that also adds the images
• We say that Airtable is the source of truth, but the actual source of truth for the website is this repo.

Improvement ideas

Some ideas on how we can improve the workflow to reduce friction:

1. Move the submission process directly to this repository. Have a PR/project.json template so that submissions can be submitted and previewed directly here. Could be combined with a tool like https://forestry.io/ for easier editing of content.
2. Have a script to sync between Airtable (using their REST API) and the files in this repository to avoid manually copying JSON. Could probably be in a GitHub action that routinely creates PRs for projects that have been approved/checked.

I'm leaning towards the first suggestion as it resolves many of the challenges mentioned above while reducing complexity significantly.

Open questions

• Do we still have a case/need for using Airtable? The main reason to use Airtable seems to be to allow non-developers to edit content. This comes with a fair bit of additional complexity which could potentially be avoided.

CC @mishmosh

[orvn]

Great points @2color. Thanks for taking the time to outline these! I've inserted a few of my thoughts below.

Do we still have a case/need for using Airtable? The main reason to use Airtable seems to be to allow non-developers to edit content. This comes with a fair bit of additional complexity which could potentially be avoided.

I'm not personally attached to it, but it depends on who will be maintaining the directories: the Airtable's original purpose was to have a user-friendly area to manage the content before shipping it to production.

Move the submission process directly to this repository. Have a PR/project.json template so that submissions can be submitted and previewed directly here. Could be combined with a tool like https://forestry.io/ for easier editing of content.

I'm on board, and I do like Forestry. One issue might be that it doesn't provide a tabular view of content and each entry requires a lot of click depth to get to.

Have a script to sync between Airtable (using their REST API) and the files in this repository to avoid manually copying JSON. Could probably be in a GitHub action that routinely creates PRs for projects that have been approved/checked.

We can also have Airtable create the PR with keys and logos, on pressing a button?

When submissions are made to Airtable, there's no easy way to ask the submitters for changes. This puts the burden of editing on us rather than being able to provide feedback and make changes in a single place.

This is one issue that can't be solved unless the Airtable base's view is somewhat public afaik. Once the Airtable form submission is made, it's immutable to the originator (this is by design, from Airtable).

[bmann]

I've used Forestry, if it can be added for edits for other elements of the site, so non Github users can do it, that would be great -- but the forms etc need a lot of work to be super usable. The team is working on a v2 product, Tina https://tina.io/ -- I haven't tried it yet.

It seems like the Airtable automation to create a PR might be the useful middle path?

And: a template could still be a good idea -- could also be an issue template? Still a manual step to include/PR/merge.

If the path is then "after you've submitted, you can make PRs to the repo" -- I don't think it's that much of a burden. For orgs that can't do this in house, they could also file an issue requesting updates. (that's still a lot of Github centric stuff).

I'm glad that I know this, because I actually have to edits to Fission's entry :)

[orvn]

The team is working on a v2 product, Tina https://tina.io/ -- I haven't tried it yet.

I've tried Tina @bmann and it's pretty great actually! It's more intended for true visual content management of marketing websites. But the real blocker is that it only works with a really specific stack (NextJS).

Not sure yet if expanding to other similar stacks is on their roadmap.

[mishmosh]

Sorry for the slow response. I think we do need to keep the data source & submission in Airtable for 2 major reasons. We recut the data in multiple ways for other reporting & analysis in the ecosystem WG.

Very supportive of improving the automation to sync data from Airtable to json, though.

[nikipdia]

Offering my help to ease anything Airtable related, if needed. There would be workarounds to get things done in Airtable (Zapier integrations, collecting data elsewhere and importing to Airtable etc.) I am not great at Airtable scripts, and write to website from Airtable though.

When submissions are made to Airtable, there's no easy way to ask the submitters for changes. This puts the burden of editing on us rather than being able to provide feedback and make changes in a single place.

You can collect info in a Google form, write an automation to create and update records in Airtable when there are new record entries and updates in the Google sheet.

[orvn]

There's an Airtable automation that's running already, it can be enhanced to push directly to this repo (a branch that would get PR'd into main).

You can collect info in a Google form, write an automation to create and update records in Airtable when there are new record entries and updates in the Google sheet.

Oh, that's a good idea. The Airtable would just need to be kept in good lockstep with the Google Form (e.g., fields like this would have to be consistent)

[nikipdia]

The Airtable would just need to be kept in good lockstep with the Google Form (e.g., fields like this would have to be consistent)
An add on can solve this problem too. It would be an opposite automation that keeps the sheet column with values picked dynamically by the form synced with those in AT. Might need some thinking how to pick all the values from AT/form entries. Probably a pivot of all the form entries.

[jamespohalloran]

The team is working on a v2 product, Tina https://tina.io/ -- I haven't tried it yet.

I've tried Tina @bmann and it's pretty great actually! It's more intended for true visual content management of marketing websites. But the real blocker is that it only works with a really specific stack (NextJS).

Not sure yet if expanding to other similar stacks is on their roadmap.

Tina Product Manager here! We're actively working on support for non-Next-based sites now actually!

[orvn]

Tina Product Manager here! We're actively working on support for non-Next-based sites now actually!

That's amazing @jamespohalloran! Thanks for the reply ❤️

[2color]

Thanks for all the input.

Based on the fact that we need to keep submissions in Airtable for the reasons Mosh stated, I would suggest the following course of action:

• Keep Airtable as the canonical source of truth
• Create a pull script to fetch the records from Airtable here. The reason for pull rather than having push automation on the Airtable side is manifold:
  • Since the goal of this automation is to update the website, it should live with the website
  • We can leverage GitHub Actions to automate as we desire while leveraging all the preexisting tooling for PR creation and workflow management.
  • We can leverage the code/sync script to also manually batch pull submissions if necessary or even detect deviations.

I would suggest avoiding introducing Google Forms/Sheets into this as it would introduce unnecessary complexity.

This still doesn't address how we provide feedback to submitters on the copy to be published besides using email. But, having gone through some of the published submissions, they don't contain that much copy, so I think we can start with the syncing and address the feedback challenge later if it's a problem.

[orvn]

Create a pull script to fetch the records from Airtable here

Yeah, that's a really good point and this is the better pattern 🤩. Github Actions should be able to accomplish this (by running a shell or node script in the CI, and adding the Airtable token as an env in Github).

I'm going to be implementing something very similar (Github → Airtable) on another PL project in the next couple of weeks and can probably port over some of that for this project.