Automatically add links to URLs in the docs site

[obulat]

Problem

The documentation site does not wrap the URLs in a link, so the URLs are not clickable. See, for example, links on https://docs.openverse.org/catalog/reference/DAGs.html

Description

We should either manually wrap the URLs with links, or add a library that would detect URLs and add links automatically. Per discussion below, we will opt to convert all of the links that are present in module docstrings to be Sphinx-compatible links. This means changing any link that exists in a module docstring in the catalog/dags/ folder from https://{link} to <https://{link}> (note the surrounding </> signs). This applies only to free-standing links (i.e. links that are not created using the markdown syntax: [example](http://{link})).

Here are some examples of the kinds of links that would need to be changed, though this list is non-exhaustive:

openverse/catalog/dags/providers/provider_api_scripts/phylopic.py

Line 9 in d80b590

Notes: http://api-docs.phylopic.org/v2/

openverse/catalog/dags/providers/provider_api_scripts/inaturalist.py

Lines 6 to 16 in 4051522

	Notes: The iNaturalist API is not intended for data scraping.
	https://api.inaturalist.org/v1/docs/
	But there is a full dump intended for sharing on S3.
	https://github.com/inaturalist/inaturalist-open-data/tree/documentation/Metadata
	Because these are exceptionally large normalized tables, as opposed to more document
	oriented API responses, we found that bringing the data into postgres first
	was the most effective approach. More detail in slack here:
	https://wordpress.slack.com/archives/C02012JB00N/p1653145643080479?thread_ts=1653082292.714469&cid=C02012JB00N
	We use the table structure defined here,
	https://github.com/inaturalist/inaturalist-open-data/blob/main/Metadata/structure.sql
	except for adding ancestry tags to the taxa table.

[AetherUnbound]

Personally I think it would be okay to add the link brackets (e.g. <link>) to all of our DAG documentation sources in the code. I just checked locally and the Airflow UI renders that as a standard link! So adding those don't affect viewing the documentation on Airflow and they improve the documentation available on our docs site, which seems like the way to go to me 🙂

On the other hand, it's not automatic so we'd need to be cognizant of any new links that get added 🤔

Regardless, we can at least handle the existing links! @obulat are you okay with me converting the body of this issue with that approach and marking it as a good first issue?

[madewithkode]

Hi @obulat @AetherUnbound This seems like a "good first issue" 😅
I'd love to work on it.

[AetherUnbound]

Sweet, thank you for your interest in contributing to Openverse! I've assigned this issue to you @madewithkode. If you have any questions, you may leave them here (or in the Slack where you've asked your other setup questions!).

Please check out our welcome and quickstart documentation pages for getting started with setting up your local environment.

[madewithkode]

Thanks @AetherUnbound . Quick question, for a change like this that doesn't really involve logic changes but can be rather observed visually, what is the checklist to be done before commit? static validation, linting, test etc? Is there a like a pre commit just recipe?

[AetherUnbound]

There is! Once you've followed the setup guide, you should be able to run just lint to perform all the static pre-commit checks for the project.

[madewithkode]

Hi @AetherUnbound, a look at sphinx's docs shows that to hyperlink a text I basically need to do something like this(example from the flickr provider script):

'<https://www.flickr.com/help/terms/api>'_

However doing this does not seem to cause the text to be hyperlinked on the generated docs, also, I'd love to know if the sphinx engine/server watches for code changes and reflects changes in real time or do I have to ctrl+c and then just documentation/live after every change? In either case though, they both seem not to work. It's my first time with sphinx so any insights would be appreciated.

[AetherUnbound]

The documentation does automatically regenerate with just documentation/live, however the DAGs.html file is generated using the just catalog/generate-docs command. If you make changes to any of the docstrings, you'll need to run that command in order to generate the new aggregated documentation. If you have the documentation server running with just documentation/live, it'll pick up the new changes after that!

I would recommend only changing the links to be <https://ww.flickr.com/help/terms/api> (without the single quotes ' or trailing underscore _). That will insure it renders in the markdown for Airflow correctly as well!

[madewithkode]

Ahhh...Thanks much for this insight. I didn't know there was an extra command to regenerate the html. I had indeed done what you suggested initially with the hyperlink syntax but because I didn't see the changes reflect I had to start tweaking to see if that was the cause lol.

[sarayourfriend]

@AetherUnbound Do you think we could still explore an automatic approach? Otherwise we should probably assume we'll need to redo #4195 once a year. If we're relying on just remembering to wrap links in brackets (or for it to be caught by manual review), it will absolutely drift and require another fix. Either we could implement a basic linting (I think Vale could actually work for this) or use/write a Sphinx plugin to transform things that look like links into links.

[madewithkode]

@sarayourfriend This sounds like a more sustainable approach, I'd be interested in working on it as well(with as much guidance as I can get ofc 😅). Let me know if it's something you're willing to delve into now or whenever it becomes priority.

P.S: I couldn't avoid using "delve" above, but I can guarantee I'm human 😅

[AetherUnbound]

I've made an issue here: #4208 - @sarayourfriend would you be able to fill in details about what the Vale approach might look like? @madewithkode you're welcome to take it on if you'd like, depending on which approach feels easier for you!

[madewithkode]

Thanks @AetherUnbound. Is it possible to sort of "reserve" this task as I am currently trying to see if I can get my hands on #3847 and possibly get to this once done with that.

[zackkrida]

Sure, @madewithkode if you want to leave a comment on #4208, we will then be able to assign it to you 😄

[sarayourfriend]

I've added a comment on #4208. We might just need to enable a MyST syntax extension.