Failed to get schema due to backend authentication

[garry-harthill-cko]

I believe the language server is reading the remote state. Remote state requires authentication.

I have a more elaborate method of authenticating against AWS which is a wrapper that rotates keys and sets environment variables, etc.

https://github.com/99designs/aws-vault

It would be a good feature if we would wrap the language servers execution in this program. Maybe a "terraform-ls.command" option so we can define the full terraform-ls command to be executed. eg:
/usr/local/bin/aws-vault exec myProfile -- terraform-ls

[radeksimko]

@garry-harthill-cko Generally the language server was designed to not require any access to credentials and other configuration that is usually provided out of bound for the exact reason that passing the right ENV variables or configuration down to Terraform is difficult from UX perspective.

The only terraform commands it runs are the following:

• terraform fmt -
• terraform version
• terraform providers schema -json

and each execution of either of these is also visible from the log (when you enable logging), e.g.

2020/06/01 08:22:00 exec.go:180: Starting /usr/local/bin/terraform ["terraform" "version"] in "/var/folders/fb/k_9ystyd08j934qcxmyv0wjc0000gp/T/"...
2020/06/01 08:22:00 exec.go:146: Waiting for command to finish ...
2020/06/01 08:22:00 exec.go:172: terraform run (/usr/local/bin/terraform ["terraform" "version"], in "/var/folders/fb/k_9ystyd08j934qcxmyv0wjc0000gp/T/", pid 19252) finished with exit code 0

2020/06/01 08:22:00 exec.go:180: Starting /usr/local/bin/terraform ["terraform" "providers" "schema" "-json"] in "/private/var/workspace/tf-test/github"...
...
2020/06/01 08:22:02 exec.go:172: terraform run (/usr/local/bin/terraform ["terraform" "providers" "schema" "-json"], in "/private/var/workspace/tf-test/github", pid 19278) finished with exit code 0

Obtaining the schema does require the workspace to be initialized, but terraform init (which would require credentials and access to the remote state) can run outside of the IDE.

I'm aware that the extension also has terraform init in the command palette - and I agree that we should reflect the reality you described there and somehow allow users to pass the configuration. That said you should be able to init outside of VSCode for the time being.

---

Do you mind sharing more details about the problem you're running into? Ideally a snippet of a log like the above?

[garry-harthill-cko]

Thanks for the quick reply.

I'm seeing this in the logs quite soon after starting vscode:

2020/06/01 10:09:43 schema_storage.go:112: error obtaining schemas: Unable to retrieve schemas: failed to get schemas: terraform (pid 17099) exited (code 1): exit status 1
stdout: ""
stderr: "\x1b[31m\n\x1b[1m\x1b[31mError: \x1b[0m\x1b[0m\x1b[1mNo valid credential sources found for AWS Provider.\n\tPlease see https://terraform.io/docs/providers/aws/index.html for more information on\n\tproviding credentials for the AWS Provider\x1b[0m\n\n\x1b[0m\x1b[0m\x1b[0m\n"

Then while editing references to a missing schema file are displayed:

2020/06/01 10:15:20 complete.go:53: HCL block found at HCL pos hcl.Pos{Line:32, Column:14, Byte:1181}
2020/06/01 10:15:20 complete.go:66: Configuration block "resource" parsed
2020/06/01 10:15:20 schema_storage.go:189: Reading "aws" resource schema
2020/06/01 10:15:20 schema_storage.go:141: Acquiring semaphore before reading schema
2020/06/01 10:15:20 rpc_logger.go:45: Error for "textDocument/completion" (ID 6): [-32098] finding completion items failed: no schema available
2020/06/01 10:15:20 server.go:204: Completed 1 requests [1.18848ms elapsed]

The workspace is initialised and all terraform actions work without errors on the command line.

Maybe I installed the extension incorrectly? I cloned into extensions directory and ran npm install.

[radeksimko]

Thank you for these useful additional details @garry-harthill-cko !

I will try to reproduce this 🔜 , but if it's a bug it would be more likely in the language server or Terraform itself, than the extension as it's the server that executes Terraform in this context. For that reason I'm going to transfer it over.

[radeksimko]

One more question - what Terraform version do you use for this configuration?

[radeksimko]

Also could you please try to run the following (in the mentioned workspace):

terraform providers schema -json

with and without your auth wrapper?

[garry-harthill-cko]

Also could you please try to run the following (in the mentioned workspace):

terraform providers schema -json

with and without your auth wrapper?

terraform -v
Terraform v0.12.25

Without the auth wrapper I get the error I described before:

Error: No valid credential sources found for AWS Provider.
        Please see https://terraform.io/docs/providers/aws/index.html for more information on
        providing credentials for the AWS Provider

With the auth wrapper it appears to work and returns a very large json document:

{"format_version":"0.1","provider_schemas":{"aws":{"provider":{"version":0,"block":{"attributes":{"access_key":{"type":"string","description":"The access key for API operations. You can retrieve this\nfrom the 'Security \u0026 Credentials' section of the AWS console.","optional":true},
...
...
}}}}}}}}

[garry-harthill-cko]

Can this schema document be downloaded separately and placed in a directory manually for use by the extension?

[radeksimko]

Thank you 🙏

It appears that terraform providers schema -json does in fact require state authentication, for some reason, which may be related to #56 🤔

You may be able to override the path to Terraform in the meantime via the following VSCode configuration (settings.json):

{
    "terraform.languageServer.args": [
        "serve",
        "-tf-exec=/auth/wrapper terraform"
    ]
}

We could also make this a more 1st-class settings in the VSCode UI in the future.

Can this schema document be downloaded separately and placed in a directory manually for use by the extension?

Not really and I don't think we'd want to support such workflow, because getting the UX right around updating it would be very tricky. Ideally the user shouldn't even need to know that there's such thing as machine-readable schema in most cases (in the happy path).

[radeksimko]

Actually the custom path flag isn't support yet, but will be in the upcoming LS version:
https://github.com/hashicorp/terraform-ls/blob/master/CHANGELOG.md#030-unreleased

[garry-harthill-cko]

I built the master branch of terraform-ls.

The -tf-exec= option does "work". However it expects a single executable I think.

I have managed to get it kind of working by calling another wrapper shell script which in turn exec's terraform with the correct environment. Not ideal.

[radeksimko]

You're right - it's not ideal.

Ideally obtaining schema shouldn't require state at all (and hence shouldn't require auth) IMO, but we can't really fix this in Terraform core retrospectively (in past versions), so we will need to have a decent solution before this is addressed in core.

However it expects a single executable I think.

Good catch! I'm not too excited about flags which expect string with spaces (or any kind of arrays), but it is a solution and think we could use something like go-shellwords to parse the string. The only caveat is that it needs some patches to work on Windows, e.g. mattn/go-shellwords#39

We can also prioritize the UI config option for setting the Terraform path in VSCode and try to construct sh -c "string" or cmd.exe /c "string" (depending on the OS) on the fly for the user, before setting it as a flag. Does that sound like a decent short-term solution?

---

(Unrelated to your use case)

There may be users which don't use wrappers like the one you mentioned, but set variables manually. I'm not sure there's any decent way we can support that, because passing environment variables between whatever process launches the IDE, the IDE, plugin system and terraform-ls is not trivial. And allowing user to set environment variables manually somewhere in settings doesn't make a great UX either, especially when they need to switch between workspaces.

Maybe we just discourage this is as a bad practice and make folks approach it either via wrappers, or by making terraform commands work out of the box (e.g. via S3 backend profile argument). 🤔

I mean even if you do store variables locally, then the sensitive ones should really be stored in something like envchain, so I'm more inclined to support that, as opposed to passing individual variables.

[paultyng]

This may just be an upstream issue in the AWS provider, see this issue among some others: hashicorp/terraform-provider-aws#13057

I misspoke, this is due to the shared auth package they use: https://github.com/hashicorp/aws-sdk-go-base

[paultyng]

Instead of taking additional arguments you could probably just write/call a shell script as the "executable" in the near term while we investigate.

[paultyng]

I think we are going to need to significantly relax env propagation and related functionality due to this backend authentication issue. See #138

[radeksimko]

I think all we need to propagate in this case is really just $PATH (and Windows variations of it)

terraform-ls/internal/terraform/exec/exec.go

Lines 24 to 32 in b6cda0d

	// Environment variables to pass through to Terraform
	var passthroughEnvVars = []string{
	// This allows Terraform to find custom-built providers
	"HOME", "USER", "USERPROFILE",
	// This allows Terraform to create crash log in the desired temp directory
	// os.TempDir would otherwise fall back to C:\Windows on Windows
	// which has no write permissions for non-admins
	"TMPDIR", "TMP", "TEMP",
	}

Then Terraform can find az and any other similar auth providers.

I would not go as far as propagating everything. It has some really bad UX implications stemming from the fact that most people launch their IDEs context/environment-less (via icon from a desktop, menu bar etc.). Even if they don't, they will need to change the environment variables when switching between workspaces, presumably without closing the IDE.

So passing through variables which are usually passed to all applications (like $PATH) makes sense, because these will most likely already be available anyway and usually don't change between workspaces, but passing variables like AWS_* and similar would lead to really bad UX.

[radeksimko]

Just to clarify though the above is a solution for #138 but not so much for this issue (#128).

That said I think that integrating go-shellwords in combination with the tf-exec flag and perhaps some UI additions on the client side might address significant number of use cases similar to what @garry-harthill-cko described above.

[danieladams456]

@radeksimko I'm just seeing this now cross linked from the other issue. I had assumed as well that terraform providers schema -json did not require state access and was static based on the providers selected during init.

What exactly is it doing with the state - some type of schema merge? Since terraform 0.13 is in beta now, is it the time to bring up making that command function independently of remote state?

[radeksimko]

What exactly is it doing with the state - some type of schema merge?

Generally a combination of config and state is needed to pick the providers before any plan or apply operation. Config may not be sufficient in cases where the user is removing resources - e.g. say you have a single aws_instance block in your config and you remove that - then Terraform still has to figure out how to remove it, even though it's not in the config anymore and therefore it probes the state which then prompts it to find aws provider.

I'm unsure what other use cases terraform providers schema -json was designed for, and so I can't comment on potential implications of changing the default behaviour, but I do think there should be a way of obtaining schema from any initd providers, regardless of config or state. That would IMO help resolve most language server use cases.

Since terraform 0.13 is in beta now, is it the time to bring up making that command function independently of remote state?

See hashicorp/terraform#24261

So yes - I think that would be probably the ideal longer-term solution, but it still raises the question of 0.12 and any earlier 0.13 releases which may not have this feature - and these versions will be around for a while, so we're hoping to find some solutions for these too, without having to backport any patches.

[piotrb]

I'm going to add a bit of complexity to this I think ..

In our config we use aws-vault (as mentioned above) together with direnv (https://direnv.net/) .. and some custom scripting to seamlessly switch between production and staging root modules ..

A simple wrapper script will not suffice here .. because I need to pass different config names to aws-vault based on if this is in a sandbox or a production sub-folder ..

perhaps something that is able to be set on a parent folder to specify the wrapper differently or something ..

Our folder structure looks like this:

shared/ <- no wrapper needed
  mod1/
  mod2/
  mod3
sandbox/ <- wrapper needs to use sandbox credentials
  root1/
  root2/
  root3/
production/ <- wrapper needs to use prod credentials
  root1/
  root2/
  root3/

[radeksimko]

@piotrb Is there a reason your custom scripting can't just take these different config names as environment variables, instead of flags and leverage direnv there, since direnv already solves the problem of environment differences between folders?

[piotrb]

direnv solves the environment issue on the shell, sure .. but the new ENV only activates when you enter the various subfolders .. so if the language server can execute stuff as a full shell, or support direnv in some way (or some other way to set env on a per folder basis) .. then this could work, a wrapper can be made easily that just changes based on the env

maybe what wasn't clear is that my 3 example root folders above are all inside one repo, so I often open it as a single project, so the LS needs to know to switch envs if its looking at a root below the different environment roots ..

[piotrb]

I suppose it might be possible to make a wrapper that executes direnv on its own .. then executes terraform wrapped in aws-vault ..

the question is .. are the changes discussed here around being able to specify a wrapper already available?

[radeksimko]

@piotrb There is currently no integration with direnv but we are considering integration of terraform-exec with direnv and there are plans for using that library in the language server at some point.

Also there is no "interpolation" of tf-exec-path - so it's currently treated as a single string argument, even if it contains spaces, so in any complex scenarios you'll need to set path to a shell script which in turn runs something like this:

direnv exec $PWD terraform $@

The idea is though that you would be able to do the above, or pass any other complex wrapper straight, once mattn/go-shellwords#39 is resolved.

[piotrb]

ok I'm the right track now .. but having one more issue now .. the wrapper requires some manually entered credentials like keychain password and OTP code .. these come as popups which is fine .. the issue is that it seems like the language server is concurrently invoked a LOT of times .. (I guess once per root) .. is there a way to make it reduce the concurrency or something? if you launch them all at once .. they all ask for password .. if you launch them in serial the first one will ask for password only ..

[radeksimko]

@piotrb That is a valid point

OTP is generally a challenge in Terraform at this point for various reasons unfortunately - it is a topic we have discussed across teams a few times, but haven't come up with a decent solution yet (read as there's no API for providers/backends for this kind of stuff).

However I have had some success using Terraform with auth tools which work in such a way that they pre-generate tokens somewhere in a known location (such as AWS STS tokens in ~/.aws/credentials) and then Terraform's provider/backend just pick it up and assume that it's always valid.

[radeksimko]

I think the ideal solution for this will really be just not requiring state for getting the schemas, and therefore not requiring auth. Technically the LSP does have methods for prompting user for input, but (a) not arbitrary input, just multiple choices, and (b) even if we were to somehow connect that to the shell executor it would probably be very fragile.

We also plan to explore some ways of obtaining the schema directly from plugin binaries via gRPC, which would sidestep this problem entirely both short term and long-term, but no work was done on that yet.

[piotrb]

tools like aws-vault do a great job of caching the credentials like you're mentioning above ..
The only issue is concurrency .. since the caching is moot if you try to launch the calls all at once .. so limiting the concurrency .. especially on the initial schema discovery .. would be ideal ..

[piotrb]

well or just not requiring authentication .. its a definitely a bit silly my editor language server needing to do authenticated calls to terraform ... it shouldn't need to do that and its a potential security risk

[radeksimko]

the caching is moot if you try to launch the calls all at once

I'm not sure I fully understand the problem here actually. 🤔 Why do concurrent calls cause a problem, if the auth is already cached for all of them? Is the cache not able to handle concurrent access?

My assumption above was that the authentication is done prior to the launch of language server, because as mentioned complex authentication requiring user input as part of the launch is not really an option anyway.

[piotrb]

You can't guarantee that the credentials are going to be current .. especially in scenarios were you are using high security setups (aws keys + OTP + assume role) (which we are) .. so this stuff can usually be cached only for ~1h .. aws-vault manages that and generally does a great job of a complex situation

The issue is that the first call after a bit of being idle will require to (at least) unlock the system keyring once again to retrieve the cached credentials (so this will pop up one authentication prompt) .. and then if the OTP is expired it will re-authenticate that too ..

So .. the first call will unlock the keychain and possibly ask for OTP .. subsequent calls have a window of time while the keychain is still open and can do stuff without any prompts ..

Of course of you parallelize the calls .. each call is actually the first .. and EVERY ONE OF THEM will pop up a keychain unlock prompt .. its a bit of a mess ;)

[radeksimko]

each call is actually the first .. and EVERY ONE OF THEM will pop up a keychain unlock prompt

Right, my point is why don't you make the first call outside of the language server, before it even launches to avoid all this? Assuming that the IDE, and therefore language server is launched by the same user, then the keychain should remain open for them, if I'm not mistaken?

As for parallelisation control - I wouldn't be opposed to making configurable as part of #186

[piotrb]

each call is actually the first .. and EVERY ONE OF THEM will pop up a keychain unlock prompt

Right, my point is why don't you make the first call outside of the language server, before it even launches to avoid all this? Assuming that the IDE, and therefore language server is launched by the same user, then the keychain should remain open for them, if I'm not mistaken?

As for parallelisation control - I wouldn't be opposed to making configurable as part of #186

Its not trivial to always make that request before its needed again and it would seriously hamper any ease at which I can just jump in and edit the files .. having module detection run doesn't provide enough benefit in order to add a whole, kind of random step to "just editing a file" .. so its easier to just let it fail in the background and make the edits needed. If it can't be automated in the tool (ie vscode/terraform-ls) its not worth the extra hassle.

[taylorsmcclure]

Yikes, this issue is causing me and my org quite a headache. We pull in our AWS creds via vault (in a wrapper) and those paths differ between projects. This makes the solution suggested earlier not work in our use-case.

[toby-w]

Can this schema document be downloaded separately and placed in a directory manually for use by the extension?

Not really and I don't think we'd want to support such workflow, because getting the UX right around updating it would be very tricky. Ideally the user shouldn't even need to know that there's such thing as machine-readable schema in most cases (in the happy path).

Hi @radeksimko @garry-harthill-cko , my organization is trying to leverage the vscode-terraform extension which leverages terraform-ls. At the moment, we are not making providers locally available nor do we have a private registry for users to get them from. I won't go into the specifics of why things are like this, but currently at best we have the output from terraform providers -json to give users. For my situation, being able to place this document to somewhere terraform-ls could load from would be best. Is there anyway for me to do this right now?

I'd like to leverage

terraform-ls/internal/schemas/preload_schemas.go

Lines 10 to 13 in c4ea46f

	pOut, vOut, err := PreloadedProviderSchemas()
	if pOut == nil || err != nil {
	return err
	}

but am unsure what format of content versions.json should have as well as where to put schemas.json.

Thanks in advance .