ssh notation support

[tongueroo]

Adds support for ssh git clone notation.

Tested several examples:

Terrafile

mod "test1", source: "git::https://github.com/boltops-tools/terraform-aws-s3"
mod "test2", source: "git::ssh://ec2-user@localhost:/home/ec2-user/environment/repo"
mod "test3", source: "git::ssh://ec2-user@localhost:environment/repo"
mod "test4", source: "git::ssh://localhost:environment/repo"
mod "test5", source: "ssh://ec2-user@localhost/home/ec2-user/environment/repo"
mod "test6", source: "ssh://ec2-user@localhost:/home/ec2-user/environment/repo"
mod "test7", source: "ssh://ec2-user@localhost:/home/ec2-user/environment/repo//subfolder"
mod "test8", source: "ssh://ec2-user@localhost:environment/repo"
mod "test9", source: "ssh://ec2-user@localhost:repo"

Also, re-tested the other sources examples on https://terraspace.cloud/docs/terrafile/sources/

Related:

• Community Post: https://community.boltops.com/t/terrafile-ssh-support/810
• Docs PR: terrafile ssh source docs terraspace-docs#27
• Docs Link: https://terraspace.cloud/docs/terrafile/sources/ssh/

[tongueroo]

Released in terraspace-bundler 0.5.0

To update:

cd terraspace-project
bundle update

Also see: https://terraspace.cloud/docs/install/updating/

[exoemay]

Hey, I still seem to have issues with this. Can it be that subdomains are not supported for the hosts?

I've tried all the methods in your examples from top to bottom.

Error messages I got:

ssh://git@src.COMPANY.com/PROJECT/REPO.git (and without.git) - /var/lib/gems/2.7.0/gems/terraspace-bundler-0.4.4/lib/terraspace_bundler/mod.rb:30:in org': undefined method split' for nil:NilClass (NoMethodError)

ssh://git@src.COMPANY.com:PROJECT/REPO - /usr/lib/ruby/2.7.0/uri/rfc3986_parser.rb:67:in split': bad URI(is not URI?): "ssh://git@src.COMPANY.com:PROJECT/REPO" (URI::InvalidURIError)`

And the version with 'git::https://....' simply states that the repository couldn't be found (it also adds a trailing slash for some reason, maybe that's the issue there). Tried it with both adding .git at the end and without.

[tongueroo]

It looks like you’re on terraspace-bundler-0.4.4. Try updating to 0.5.0.

bundle update
bundle info terraspace
bundle info terraspace-bundler

You should see 0.5.0. Then

terraspace bundle

[exoemay]

Hm... I created a fully new docker image of TS 1.0.1 and ran a 'bundle update' as well. But yes, I didn't verify the bundler version. Let me do that real quick.

[tongueroo]

Oh interesting. Released terraspace-bundler 0.5.0 just before terraspace 1.0.1, so the build process should had use the latest terraspace-bundler. But since the build process now runs upon rubygems webhooks, it might be too fast and it installed terraspace-bundler 0.4.4 since rubygems bundler CDN can take 20m to update. That’s a guess. Will have to dig into more.

[exoemay]

I ran the new build this morning right after seeing the news (so around 5 hours ago). But yes, it's 0.4.4 unfortunately. I will run another docker build and see if this works. we bake the bundle install/update into the docker image to have fixed dependencies, I guess it still grabbed the old version. I'll get back to you after the next run.

[exoemay]

I think I found the issue:

You pinned the version of 'terraspace-bundler' in terraspace via ~> on the patch part of the version. Therefore 0.5.0 couldn't be used:

Bundler could not find compatible versions for gem "terraspace-bundler":
  In Gemfile:
    terraspace-bundler (>= 0.5.0)

    terraspace (= 1.0.1) was resolved to 1.0.1, which depends on
      terraspace-bundler (~> 0.4.4)

[tongueroo]

Cool. Glad to hear it’s been resolved. Thanks for the update. 🎊

[exoemay]

Well it's not really resolved. You would need to update the terraspace gem versioning itself for it to be fixed :)

Bundle is refusing to install terraspace-bundle in anything higher than 0.4.4 right now.

[tongueroo]

🤦🏻‍♂️ Literally, realized that it was terraspace.gemspec that you're talking about vs the project's Gemfile.lock after posting the comment. 🤣 Ok. Fixed in boltops-tools/terraspace#190

[exoemay]

Thank you, that was very quick! :) So will this be available in 1.0.2 then?

[tongueroo]

Yup. Already released in 1.0.2

[exoemay]

Great! I will let you know if I see any more issues. You've been a tremendous help in this, much appreciated!

[exoemay]

Hm... are you sure you are actually doing the clone via ssh? It seems you strip the ssh:// information from the git clone url, which means git will try https instead.

source: 'ssh://git@src.COMPANY.com:PROJECT/REPO'

Bundling with Terrafile...
ERROR: git clone git@src.COMPANY.com:PROJECT/REPO 2>&1
Cloning into 'REPO '...
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The expected 'git clone' command should have been: git clone ssh://git@src.COMPANY.com/PROJECT/REPO

Not sure about the colon between host and org/project, but the ssh:// protocol needs to be specified.

[tongueroo]

RE: Not sure about the colon between host and org/project

So believe the separator between the host and path makes a difference.

• When the separator is a /, then the ssh:// is kept, but then the path must be an absolute path.
• When the separator is a : then the ssh:// is removed and relative paths can be used.

This stack overflow helped figure that out: https://stackoverflow.com/questions/6167905/git-clone-through-ssh/16134428#16134428

Code:

terraspace-bundler/lib/terraspace_bundler/mod/props.rb

Lines 68 to 111 in 4f9b4b5

	# git_source_url is normalized
	#
	# See: https://stackoverflow.com/questions/6167905/git-clone-through-ssh
	#
	# ssh://username@host.xz/absolute/path/to/repo.git/ - just a forward slash for absolute path on server
	# username@host.xz:relative/path/to/repo.git/ - just a colon (it mustn't have the ssh:// for relative path on server (relative to home dir of username on server machine)
	#
	# When colon is separator for relative path do not want ssh prepended
	#
	# git clone ec2-user@localhost:repo
	#
	# When slash is separator for absolute path want ssh prepended
	#
	# git clone ssh://ec2-user@localhost/path/to/repo => valid URI
	#
	def git_source_url
	if @source.include?('http') || @source.include?(':')
	# Examples:
	# mod "pet1", source: "https://github.com/tongueroo/pet"
	# mod "pet2", source: "git@github.com:tongueroo/pet"
	# mod "pet3", source: "git@gitlab.com:tongueroo/pet"
	# mod "pet4", source: "git@bitbucket.org:tongueroo/pet"
	# mod "example3", source: "git::https://example.com/example-module.git"
	#
	# sub to allow for generic git repo notiation
	source = @source.sub('git::','')
	if source.include?('ssh://')
	if source.count(':') == 1
	return source
	else
	return source.sub('ssh://', '')
	end
	end
	source
	else
	# Examples:
	# mod "pet", source: "tongueroo/pet"
	# Or:
	# org "tongueroo"
	# mod "pet", source: "pet"
	org_source = @source.include?('/') ? @source : "#{TB.config.org}/#{@source}" # adds inferred org
	"#{TB.config.base_clone_url}#{org_source}"
	end
	end

It’s one of the many tricky edge cases that ran into with this. Wrote specs to cover it.

terraspace-bundler/spec/terraform_bundler/mod/props_spec.rb

Line 149 in 4f9b4b5

context "ssh without colon supports only absolute path source" do

So believe that's how it should work. 🧐

The interesting thing is that it looks like the clone is not working with the colon separator for your source or maybe setup.

git clone git@src.COMPANY.com:PROJECT/REPO

Wondering if it's a git version thing. FWIW, tested with these versions of git

git version 2.23.4
git version 2.31.0

Unsure why the git source with the colon : separator doesn't work for you. It’s a bummer. 😕Maybe try the slash / separator and specifying the full path.

Also, maybe try cloning with the colon : separator directly without terraspace bundle as a test. Seeing if can figure out the reason why that's not working. 👍

[exoemay]

Clone isn't working without the ssh:// on our bitbucket server as the https port is restricted to browsers only for security reasons. I will try the absolute path. Do you know if that does that include the .git ending or not?

We do a regular git clone from the same repository a line above the "terraspace bundle" as a double check the entire time to be sure. But we have to include the ssh protocol specifier for it to work.

I will get back to you again :) thanks for the explanations as well.

[exoemay]

Okay. After figuring out that half my problems were due to the fact that git doesn't like to run in root in a docker image (you probably want to add a user to your docker images as well, otherwise they won't run in a ci/cd pipeline like on Jenkins) I finally got it to work. Your changes work fine :) Thank you!

[tongueroo]

One note, was about to add to the terraspace docs about possible issues using the root user, but tested it in a Docker container and was able to git clone with the root user. Example:

root@6ad0a978c83a:~/infra# whoami
root
root@6ad0a978c83a:~/infra# git clone user@src.example.com:environment/repo                    
Cloning into 'repo'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 9 (delta 1), reused 0 (delta 0)
Receiving objects: 100% (9/9), done.
Resolving deltas: 100% (1/1), done.
root@6ad0a978c83a:~/infra# 

Also, the terraspace bundle worked when the user was root also.

root@6ad0a978c83a:~/infra# cat Terrafile
mod "mod1", source: "ssh://user@src.example.com:environment/repo"
root@6ad0a978c83a:~/infra# terraspace bundle
Bundling with Terrafile...
Exporting mod1
Modules saved to vendor/modules
root@6ad0a978c83a:~/infra# 

Just noting for posterity. Glad to hear that it worked out.

[exoemay]

It might run on your local machine, but it will not run in any ci/cd, as those usually enforce using uid 1000 in a docker container for security reasons. Jenkins does it, Bamboo does it and I'm pretty sure you get the same effect in Github Actions (but I'm not able to test this atm).

The only thing necessary for us was making sure we created a user on the docker image. The dockerfile didn't have to run it in at all and all installation was done using root in the Dockerfile.