-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot download private git repo with artifact stanza #2818
Comments
While digging into unit tests of go-getter I found that the right (probably) way to get private git repo is
BUT
|
is the sshkey value base64 encoded?
|
Yes. It is. |
I am really confused now. I looked into 0.5.6 release code and found that URLs and options should be interpolated:
func getGetterUrl(taskEnv *env.TaskEnvironment, artifact *structs.TaskArtifact) (string, error) {
taskEnv.Build()
source := taskEnv.ReplaceEnv(artifact.GetterSource)
// Handle an invalid URL when given a go-getter url such as
// git@github.com:hashicorp/nomad.git
gitSSH := false
if strings.HasPrefix(source, gitSSHPrefix) {
gitSSH = true
source = source[len(gitSSHPrefix):]
}
u, err := url.Parse(source)
if err != nil {
return "", fmt.Errorf("failed to parse source URL %q: %v", artifact.GetterSource, err)
}
// Build the url
q := u.Query()
for k, v := range artifact.GetterOptions {
q.Add(k, taskEnv.ReplaceEnv(v))
}
u.RawQuery = q.Encode()
// Add the prefix back
url := u.String()
if gitSSH {
url = fmt.Sprintf("%s%s", gitSSHPrefix, url)
}
return url, nil
} Either I am doing something wrong or my environment variables are not in taskEnv at the moment of this function execution. |
I am able to pull using this job template
However, I can't use vault template stanza to pull the repo. Enter passphrase for key '/tmp/go-getter336367246': 2017/07/17 23:14:48.124869 [DEBUG] http: Request /v1/jobs?prefix=docs (1.425495ms) |
@srivignessh Can you show me what you mean "use vault template stanza". Not sure I know what you mean. |
This fails.
Vault 'secret/key' directory is stored the value="base64 string", Vault policy is appropriately configured. It fails with this error |
@srivignessh Ah, yes that is not supported. The content of a file is not read. There is an issue tracking using Vault values in the job file which is what you really need. For the time being the sshkey has to be in plaintext in the file. |
Please update the issue when it clears. |
I'm also unable to get artifact from private Git repository, even when putting base64 encoded, unencrypted private key. I'm getting errors like this:
I believe it's related to hashicorp/go-getter#55 i.e. github.com is not a known host, so ssh client refuses to open connection. How can I work around it? |
The workaround is to call
on every node running a Consul client. Not very pretty. Also, putting literal key in a job file is not very pretty. Is there any way to avoid it? It seems that interpolations just don't work for
Below is a feature request. It would be perfect if
The way it is right now is really cumbersome and insecure. I believe this could be implemented without changes to go-getter. Nomad could read the file and encode it with base64 before passing it to go-getter. But changing go-getter to accept the file in the URL would be even better, as the risk of leaking the key to logs would be reduced. |
ideally we could hit the vault pki backend to issue the private key. For existing keys outside of vault wouldn't it be even easier to skip rendering the template and just pass the vault secret directly to the options? Something like:
|
Hey there Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this. Thanks! |
+1 |
Couldn't get this to work for bitbucket and gitlab projects(Probably because they use dynamic IPs) . This seems like a major ux issue for new users and documentation around this is shallow. Giving up on using git with nomad for now, will try things with containers. |
+1 |
We could use this as well :-) |
+1 |
Downloading a specific file from a private github repo has been problematic for me, especially on Windows hosts that run Nomad. Doing the "ssh-keyscan -H github.com" implies every jobspec that needs to pull files from a private github repo need to have an additional task in the taskgroups. My workaround has been to run the Python project "githubdl" in a task. This works on both Windows and Linux Nomad hosts. One bonus is that it's a lot easier to see exactly what's going on when troubleshooting jobspecs that need to download something from a private Github repo. You have to add some additional consideration in the task that executes the file that's being downloaded. I can see the difficulty in this particular kind of use case, so I figured I'd mention a workflow that might help. Here's the githubdl project: https://pypi.org/project/githubdl/ |
+1 |
Same, trying to download from private github repo. Looking at node's logs there is |
+1 |
on windows hosts, this is still troublesome..
|
Going to close this issue out. #10036 (comment) has examples of this working, but the ssh-keyscan is still a necessary step (ref hashicorp/go-getter#55). This is by design as we should not be automatically accepting host keys. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.5.6
Operating system and Environment details
Client running in Docker (17.06 CE):
Linux b82934a8bd68 4.8.0-1-amd64 #1 SMP Debian 4.8.7-1 (2016-11-13) x86_64 x86_64 x86_64 GNU/Linux
Server running agent and nodes:
Linux my-hostname 4.8.0-1-amd64 #1 SMP Debian 4.8.7-1 (2016-11-13) x86_64 GNU/Linux
Issue
Nomad or maybe go-getter (cannot check) fails when specifying private git repo in artifact stanza
Reproduction steps
Expected result
Actual result
Job file
The text was updated successfully, but these errors were encountered: