Clean up API URL environment variables

[sarayourfriend]

Current Situation

As part of the API migration to openverse.org we will need to have a clean switch over of the "canonical" URL for the application from the .engineering domain to the .org domain. However, the current API configuration does not make this easy, with many hard-coded references to the .engineering domain.

Suggested Improvement

We should consolidate the API domain information into a set of two new environment variables, with all other related domains derived from these two:

• CANONICAL_DOMAIN: str
• ALTERNATIVE_DOMAINS: list[str]

We should then make the following changes:

• Replace all usages of BASE_URL with CANONICAL_DOMAIN and remove BASE_URL
• CSRF_TRUSTED_ORIGINS set to [BASE_URL] + [f"https://{domain}" for domain in ALTERNATIVE_DOMAINS]
• ALLOWED_HOSTS Django setting to use [CANONICAL_DOMAIN] + ALTERNATIVE_DOMAINS instead of reading from ALLOWED_HOSTS environment variable

Replace all hard references to api.openverse.engineering in the documentation to CANONICAL_DOMAIN.

We can do the following cleanup as well:

• Remove LOAD_BALANCER_URL usage. See https://github.com/WordPress/openverse-infrastructure/pull/804 for rationalisation, where the environment variable is removed entirely
• Related to that, remove USE_X_FORWARDED_HOST, which enables a header we never use

Benefit

Reduce the number of changes necessary to change the "canonical" domain of the API from the .engineering domain to .org.

Additional context

There are two things to consider with these changes: how to deploy them directly, and then how to use these new variables during the migration.

To deploy these specific changes:

• First add these new environment variables to the live API environment variables, without removing BASE_URL and ALLOWED_HOSTS environment variables.
• Deploy the API-side code changes to use the new variables and ignore the old ones
• Remove the old environment variables from the live API configuration

This follows our basic zero-downtime approach to environment variable management.

To use these variables during the migration:

• When enabling the .org domain as a new location, before making it the "canonical" URL, add it to ADDITIONAL_DOMAINS.
• After confirming everything works as expected, and before enabling the redirect from the .engineering domain, switch CANONICAL_DOMAIN to the .org domain, and updating additional domains to have the old canonical domain.
• After we enable the hard redirect from the .engineering domains, remove them from additional domains list, leaving only any alternative .org domains. For staging, we will only have api-staging.openverse.org as the canonical domain, and no additional domain. For production, we'll have api.openverse.org as the canonical domain, and api-production.openverse.org as an additional domain.