List available datasets in Gatsby (not Auspice)

[jameshadfield]

We have begun building out pages and components which render a filterable list of datasets / narratives, such as /influenza and /sars-cov-2. (Note the former may be moved to "/flu" shortly.) These will be expanded in due course to cover different “sources” such as nextstrain community, nextstrain core, nextstrain groups. This is detailed in #305.

As part of this, we wish for URLs which are not backed by a dataset to render / redirect a corresponding “parent” page. For instance, /ncov/antarctica should redirect to /sars-cov-2. (This issue use the word “redirect” but it doesn’t have to be client-redirects, it would also be possible to maintain the URL and simply render the page of /sars-cov-2.)

What happens currently

Most URLs end up in Auspice (see flowchart) where a request is made for the corresponding dataset. If this dataset doesn't exist, Auspice shows a 404 page and a listing of datasets/narratives for that Source (core, staging etc). Note that this page is actually defined in nextstrain.org as a build-time customisation to auspice.

What is desired

Given a path X without an associated dataset, we wish to redirect to the closest “parent” page. Examples:

1. We maintain a list of core dataset prefixes. Any path matching these, but not backed by a dataset, should be redirected to /pathogens (or /core) which lists all core datasets.
2. Similarly the “staging” datasets, for which we don’t maintain a list of prefixes, should redirect to /staging.
3. Pages starting with /flu or ncov, /influenza, /sars-cov-2 etc should redirect to /influenza or /sars-cov-2 as appropriate. In the future there may be more pages such as these.
4. Groups URLs which don’t define a valid group (or aren’t authorised for that group) go to /groups. These should display all groups datasets, including private ones (see Routinely collect shared datasets & narratives #306).
5. Groups URLs for which a dataset doesn’t exist go to /groups/name which is rendered from within Gatsby, not Auspice.
6. Community URLs for which a dataset doesn’t exist go to /community/org/name which is rendered from Gatsby, irregardless of whether a default dataset exists.
7. Similarly to (6), /community/org/name is a Gatsby page if no default dataset is present. This allows us to remove the “special cased” 204 response from the getDataset API.
8. These pages are all rendered by Gatsby using reusable components, and we should expose the fact that the URL here wasn’t

Complexities
There are two major ones that I am aware of:

• We currently don't know if a dataset exists until we try to access it. If it 404s then we can assume it doesn't currently exist. We must recheck this each time as new datasets may have been added (or removed) since the last check.
• The /community/$org/$repo address should display a different page (one from Gatsby, one from Auspice) depending on whether a default community dataset exists. (See 6 and 7 above)

[jameshadfield]

#311 began an implementation of this, which is currently viewable in the next branch

[jameshadfield]

Most listings are now performed by Gatsby code 🙌

Remaining tasks:

• /fetch URLs can still end up at the auspice splash page
• URLs containing "narrative" can still end up at the auspice splash page
• remove the customised auspice splash page which can no longer be accessed