Different behavior between r10k and g10k when referencing Puppetforge module with git address

[mikey179]

I just stumbled about a difference between r10k and g10k. Given the following definition in my Puppetfile:

mod 'mayflower/php',
  :git => 'https://github.com/mayflower/puppet-php.git',
  :tag => '3.4.2'

With g10k the following error appears:

2015/11/10 15:56:56 downloadForgeModule(): Unexpected response code while GETing https://forgeapi.puppetlabs.com/v3/files/mayflower-php-:git => .tar.gz400 Bad Request

With r10k instead, it will create simply drop the mayflower/ part and create the folder php where it stores the module in.

mayflower/php is the official name of the Puppetforge module which I'd like to use for reference purpose, but as Puppetforge modules are sometimes out of date I like to reference the Github repository with the tag directly. Or, people use the forge module first and switch to the Github repo later but don't rename it in the Puppetfile and might run into this problem then.

[xorpaul]

I acknowledge that this is a bug that does make g10k incompatible with r10k.

My problem is that I am declaring a module a Puppetlabs Forge module if I encounter the owner/modulename syntax (

g10k/config.go

Lines 89 to 90 in cf471fd

	reForgeModule := regexp.MustCompile("^\\s*(?:mod)\\s*['\"]?([^'\"]+/[^'\"]+)['\"](?:\\s*(,)\\s*['\"]?([^'\"]*))?")
	reGitModule := regexp.MustCompile("^\\s*(?:mod)\\s*['\"]?([^'\"/]+)['\"]\\s*,(.*)")

)

That's why g10k is trying to fetch something from the forge API in this case.

What I need to do to support the owner/modulename syntax for Git modules is to completely rewrite the Puppetfile config parsing, which is currently done with extremely ugly regexes. Start differentiating between Git and Puppetlabs Forge modules by looking at the given attributes.
If :git then it's definitely a Git module so remove "owner/" and so on.

I'll try to do that in https://github.com/xorpaul/g10k/tree/fix_modulename

The fact that the Puppetfile syntax is horrible to parse with anything except ruby where you can use the puppet library parser and the high maintenance/debug cost that this implies does make want to reject this bug and putting my effort into supporting a standard data format like JSON, YAML, TOML etc instead.

For now please use mod 'php' for your Git module and we will see if I can rework the config parsing in the near future with justified effort.

[ghost]

The "namespacing-feature" in r10k when handling Forge modules is not something that should be re-implemented when it comes to GIT, as the underlying reason for said namespacing does not exists there, afaik.

Forge: Modules have to be addressed as "maintainer/module name", since the maintainer part of this is the only available differentiator when querying the forge - "team1/ntp" vs. "team2/ntp". So the namespacing there is fixed.
Git: With GIT it's quite a different story, there is no such definition central to GIT that would lead to such a rigorous, predictable namespacing.
With something like gitlab you have /GitlabGroup/Repo.git - with other git providers you do not necessarily have that namespacing at all, so there the mapping into "maintainer/module" would make no sense.

Point being that carrying this over 1:1 from the Forge might be useful to some incarnations of git services (and even there only if the Puppetfile capabilities get vastly expanded upon), but definitely not all, so this can't be a "general g10k/GIT" feature, but a specific one for "g10k/gitlab | g10k/github".

If this got implemented with no additional features around this, this does not seem to actually serve any purpose beyond keeping the status as a "drop in" replacement for r10k.

In r10k this actually leads to bad/buggy behaviour right now - you can make r10k merge 2 different modules with the same name from different owners into the same target module directory, simply by combining in the same Puppetfile
ForgeModule "team1/ntp" and
GITModule "team2/ntp"
This will create 1 ntp module dir with mashed up module content from both sources.
Not sure this should be emulated here, just for consistency? ;)

[mikey179]

Well, indeed it doesn't seem to be a good idea to be consistent just for the sake of consistency when this results in such a bad merge behavior. However, I think a better error message than the current one would definitely be helpful, e.g. pointing out that the use of "maintainer/modulename" is not supported with Git addresses. But this still leaves the problem with syntax and regular expressions. ;)

[xorpaul]

Yeah, I will definitely add better error handling for this.

[blackknight36]

Puppet best practice is that all modules are named $org-$module, even for internal modules. You will see this if you run the metadata-json-lint on one of your modules. This bug makes g10k unusable and it is not a "drop-in" replacement. The documentation should be updated to reflect this.