GIT_PROVIDER_CONFIG required placeholders, eg: {namespace}

[freakinhippie]

I'm curious about the reasoning behind the checks in terrareg/models.py that confirm that the {namespace} placeholder is present in each of the URLs in the GIT_PROVIDER_CONFIG setting.

For example, if I run this:

docker run -ti -p 5000:5000 \
  -e PUBLIC_URL=http://localhost:5000 \
  -e MIGRATE_DATABASE=True \
  -e SECRET_KEY=$SECRET_KEY \
  -e ADMIN_AUTHENTICATION_TOKEN=MySuperSecretPassword \
  -e GIT_PROVIDER_CONFIG='[{
    "name": "Github Custom",
    "base_url": "https://github.com/my_org/{module}",
    "clone_url": "ssh://git@github.com/my_org/{module}.git",
    "browse_url": "https://github.com/my_org/{module}/tree/{tag}/{path}"
  }]' ghcr.io/matthewjohn/terrareg:latest

The app produces the following error:

Traceback (most recent call last):
  File "/app/./terrareg.py", line 20, in <module>
    s = Server(ssl_public_key=args.ssl_pub_key, ssl_private_key=args.ssl_priv_key)
  File "/app/terrareg/server/__init__.py", line 155, in __init__
    terrareg.models.GitProvider.initialise_from_config()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/terrareg/models.py", line 503, in initialise_from_config
    GitUrlValidator(git_provider_config['base_url'] + git_path_template).validate(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        requires_namespace_placeholder=True,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        requires_path_placeholder=False
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/app/terrareg/validators.py", line 38, in validate
    raise RepositoryUrlParseError('Namespace placeholder not present in URL')
terrareg.errors.RepositoryUrlParseError: Namespace placeholder not present in URL

This prevents me from being able to define a namespace in Terrareg that doesn't align with the repo URL.

It seems to me that the URLs should simply use the placeholder if it is defined without the validation, but as the validation is there I have to assume there is a good reason for it. And I didn't want to raise an issue for it without understanding the background.

[MatthewJohn]

Hey @freakinhippie ,

Yes and no.
It's sort of just been iterations from Hashicorp's implementation.
If we take their registry then we say: / (or provider) == github.com//terraform-provider- or the module varient.
So then, the progression is sort of

• Wanting to support alternatives to github
• Wanting to support hosting in slightly alternative paths (different naming scheme for modules/providers etc)

The main reason I imagine that they're all enforced is to try to avoid conflicts.
So, if we take namespace out of the URL, then creating two namespaces with the same module will result in the same URL.

But, I'm not massively commited either way :) If there's a useful situation without the namespace, then it makes sense to remove.
The only concern I might have is that pumping some template into an env variable with little-to-no feedback except for trying to index a module sort of sucks. So having validation felt like a good way to do this (in a sense). It would of course be better to display the resulting URLs inside the module provider settings somewhere.

[freakinhippie]

I recognize that the Hashicorp Terraform Registry has some constraints that appear to have been adopted by terrareg, such as the repo naming for providers is {namespace}/terraform-provider-{name} and modules is {namespace}/terraform-{provider}-{name}. Unfortunately, my use case is perhaps out of the ordinary.

For a time, my organization self hosted an internal SCM system which mapped neatly to the {namespace} requirement, where internal teams more or less owned their namespace. However, recently we've started a migration to another SaaS SCM provider and with that our ability to segment by namespace was lost. So, many repositories are being renamed to include a sort of namespace repository prefix, think team1-{repo}, team2-repo, etc. However, the crux is that not all repos include this prefix, and some don't align with the existing namespaces defined in Terrareg. So, I was hoping that I could create multiple providers that remapped the repos.

For example, using Github as the example SaaS offering since that's what Hashicorp TFR expects, suppose these modules and corresponding git provider configs exist:

NOTE: please ignore that each team has a vpc module as this is a contrived example for illustrative purposes.

Repo URL	Namespace (TFR)	Module	TF Provider	Git Provider ID
https://github.com/my_org/terraform-aws-vpc	platform	vpc	aws	1
https://github.com/my_org/secops-terraform-aws-vpc	sec	vpc	aws	2
https://github.com/my_org/team1-terraform-aws-vpc	team1	vpc	aws	3
https://github.com/my_org/team2-terraform-aws-vpc	team2	vpc	aws	3

GIT_PROVIDER_CONFIG:

[
  { /* ID=1 */
    "name": "Github",
    "base_url": "https://github.com/my_org/{module}",
    "clone_url": "ssh://git@github.com/my_org/{module}.git",
    "browse_url": "https://github.com/my_org/{module}/tree/{tag}/{path}"
  },
  { /* ID=2 */
    "name": "Github [ns:sec]",
    "base_url": "https://github.com/my_org/secops-{module}",
    "clone_url": "ssh://git@github.com/my_org/secops-{module}.git",
    "browse_url": "https://github.com/my_org/secops-{module}/tree/{tag}/{path}"
  },
  { /* ID=3 */
    "name": "Github [ns:team]",
    "base_url": "https://github.com/my_org/{namespace}-{module}",
    "clone_url": "ssh://git@github.com/my_org/{namespace}-{module}.git",
    "browse_url": "https://github.com/my_org/{namespace}-{module}/tree/{tag}/{path}"
  }
]

I thought that I could address my namespace mapping/collision problems with this sort of setup, but then I encountered the error mentioned above. Subsequently when looking at the code it appears that the repo prefixes of terraform-provider- and terraform- are hard-coded in many places.

I don't know if there is an appetite to address this limitation, or if the existing Terrareg user base (besides myself) would benefit from it. I could imagine adding another placeholder such as {prefix} which maps to terraform-provider- or terraform- as needed, and then relying on the git provider to define the full URL, like so:

[
  { /* ID=1 */
    "name": "Github",
    "base_url": "https://github.com/my_org/{prefix}{module}",
    "clone_url": "ssh://git@github.com/my_org/{prefix}{module}.git",
    "browse_url": "https://github.com/my_org/{prefix}{module}/tree/{tag}/{path}"
  },
  { /* ID=2 */
    "name": "Github [ns:sec]",
    "base_url": "https://github.com/my_org/secops-{prefix}{module}",
    "clone_url": "ssh://git@github.com/my_org/secops-{prefix}{module}.git",
    "browse_url": "https://github.com/my_org/secops-{prefix}{module}/tree/{tag}/{path}"
  },
  { /* ID=3 */
    "name": "Github [ns:team]",
    "base_url": "https://github.com/my_org/{namespace}-{prefix}{module}",
    "clone_url": "ssh://git@github.com/my_org/{namespace}-{prefix}{module}.git",
    "browse_url": "https://github.com/my_org/{namespace}-{prefix}{module}/tree/{tag}/{path}"
  }
]

Unfortunately, this suggestion is just a half-baked solution at best.

[MatthewJohn]

Subsequently when looking at the code it appears that the repo prefixes of terraform-provider- and terraform- are hard-coded in many places.

I'm a little uncertain about this - for providers, yes terraform-provider- is currently enforced, to match the Hashicorp completely (initially). But for modules, there is no such limitation, you are free to put whatever you want.
So, if we ignore the namespace situation (which can probably be relaxed), I assume yours would look like:

[
  { /* ID=1 */
    "name": "Github",
    "base_url": "https://github.com/my_org/terraform-{provider}-{module}",
    "clone_url": "ssh://git@github.com/my_org/terraform-{provider}-{module}.git",
    "browse_url": "https://github.com/my_org/terraform-{provider}-{module}/tree/{tag}/{path}"
  },
  { /* ID=2 */
    "name": "Github [ns:sec]",
    "base_url": "https://github.com/my_org/secops-{provider}-{module}",
    "clone_url": "ssh://git@github.com/my_org/secops-{provider}-{module}.git",
    "browse_url": "https://github.com/my_org/secops-{provider}-{module}/tree/{tag}/{path}"
  },
  { /* ID=3 */
    "name": "Github [ns:team]",
    "base_url": "https://github.com/my_org/{namespace}-{provider}-{module}",
    "clone_url": "ssh://git@github.com/my_org/{namespace}-{provider}-{module}.git",
    "browse_url": "https://github.com/my_org/{namespace}-{provider}-{module}/tree/{tag}/{path}"
  }
]