internal/service: add support for didChangeWatchedFiles

[danishprakash]

Closes #15

[hashicorp-cla]

All committers have signed the CLA.

[radeksimko]

Hi @danishprakash
Sorry for the delay in reviewing this PR.

There is a few PRs I'd like to merge first before looking at this one. Specifically #839 and #843

Thank you for the patience.

[radeksimko]

With regards to the implementation and to answer some of your questions above:

I'd have to test this for real but my assumption is that workspace/didChangeWatchedFiles is perceived more like a "trigger" notification, primarily for files which are not open. Files which are open would presumably still receive textDocument/didChange with the new content. Closed files are just assumed to be saved to disk by the time we receive the workspace/didChangeWatchedFiles notification - so we shouldn't need to be updating anything internally.

So with that in mind I think that the implementation could be simplified to basically this

terraform-ls/internal/langserver/handlers/did_change.go

Lines 55 to 60 in 8529ae5

	jobIds, err := svc.parseAndDecodeModule(dh.Dir)
	if err != nil {
	return err
	}
	return svc.stateStore.JobStore.WaitForJobs(ctx, jobIds...)

but of course we'd first have to filter/deduplicate it down to just directories (module paths) and decide what do we do with files which the server doesn't treat as editable (i.e. anything other than *.tf or *.tfvars). I think we could probably just ignore and raise warnings for those - as per docs clients aren't supposed to send it to server anyway.

[radeksimko]

@danishprakash The mentioned PRs were both now merged. When you have a moment - do you mind rebasing your PR?

Let me know if you need any further help beyond the answers I provided.

[danishprakash]

@radeksimko thanks for the explanation. That helped a lot, however, I still have a few doubts/questions about some of those points, please correct me If I'm wrong or point me to the right resource If I seem to have missed something but basically

workspace/didChangeWatchedFiles is perceived more like a "trigger" notification, primarily for files which are not open.

I understand we have textDocument/didChange for file changes but the spec doesn't mention the above explicitly, right? It simply states the following:

The watched files notification is sent from the client to the server 
when the client detects changes to files and folders watched by the 
language client

Closed files are just assumed to be saved to disk by the time we receive the workspace/didChangeWatchedFiles notification - so we shouldn't need to be updating anything internally.

Again, I might be missing something here but based on the spec, can we say the above is an assumption at best?

Finally, I'm also assuming based on the discussion above that we're going to ignore CREATE type event since we're receiving workspace/didCreateFiles event with the required metadata?

[radeksimko]

You are right in that there are unfortunately parts of LSP which are not explicitly documented within the spec but servers pretty much rely on certain client implementations, most often the VS Code client library. The assumptions I made above are based on the VS Code client library's behaviour.

In general the spec doesn't seem to provide a lot of details on what the server should do with received documents from textDocument/didOpen and other text sync notifications, but AFAICT most servers do what this LS does - correct me if I'm wrong.

All I can find is this

The document open notification is sent from the client to the server to signal newly opened text documents. The document’s content is now managed by the client and the server must not try to read the document’s content using the document’s Uri.

and "read the document’s content using the document’s Uri" I'd interpret as "read the document's content from the disk/filesystem".

The spec also doesn't explicitly call out how remote environments should be handled on either client or server side. I'm guessing the assumption is mostly that LSP generally won't be used in a remote environment but I haven't researched this too much.

If you have some other assumptions/interpretations mismatching mine I'd be happy to discuss them! 😄 Relatedly I'm sure the LSP folks would welcome PRs with clarifications at https://github.com/microsoft/language-server-protocol

I'm also assuming based on the discussion above that we're going to ignore CREATE type event since we're receiving workspace/didCreateFiles event with the required metadata?

That would be my assumption as well. The only edge case I would check however is how renames/moves are handled by the client - for example:

• /tmp/alpha/main.tf is open
• /tmp/alpha/main.tf is moved to /tmp/beta/main.tf & /tmp/charlie/main.tf is moved to /tmp/alpha/main.tf
• assuming the rename is sent as part of the same single notification, this could either be 1 Change event or multiple Create and Delete events

btw. when the spec nor the VS Code client implementation provides us sufficient clarity, we consult the gopls codebase. A quick search through their codebase suggests they make similar assumptions: https://github.com/golang/tools/blob/e998cd2c59314075639579879898df1c61b57e2e/internal/lsp/cache/view.go#L441-L446

They also seem to implement a delay (default 100ms) and process notifications in batches, but I wouldn't implement it here yet unless we have more context and real evidence it affects our users too.
https://github.com/golang/tools/blob/78ff15e68069f59866a005585edc7469f8671fc0/gopls/doc/settings.md#experimentalwatchedfiledelay-timeduration

[radeksimko]

This is getting quite close. I left you some comments in-line.

Also feel free to checkout this PR branch if you need to test it: hashicorp/vscode-terraform#1095

[radeksimko]

@danishprakash I will take this over to bring it over the finish line (while naturally retaining your authorship for existing commits), I hope you don't mind.

[danishprakash]

@radeksimko sure, but if it's okay with you, I'd love to know what all is pending and required If I were to complete this PR? I know tests are one. Apart from that, I'd really like to understand the complications if there are any. Thanks :)

[radeksimko]

@danishprakash Aside from the tests, which I added in the 2nd commit, the first one addresses the following:

• Change two UI-visible warnings into logs as per conversation above. User is less likely to be able to address the two cases. Nudging users to report bugs is what I would consider good practice though as it gives us the opportunity to fix things.

  • terraform-ls/internal/langserver/handlers/did_change_workspace_folders.go

    Lines 24 to 29 in 4e35a7d

    	jrpc2.ServerFromContext(ctx).Notify(ctx, "window/showMessage", &lsp.ShowMessageParams{
    	Type: lsp.Warning,
    	Message: fmt.Sprintf("Ignoring removed workspace folder %s: %s."+
    	" This is most likely bug, please report it.", removed.URI, err),
    	})
    	continue

  • terraform-ls/internal/langserver/handlers/did_change_workspace_folders.go

    Lines 60 to 65 in 4e35a7d

    	jrpc2.ServerFromContext(ctx).Notify(ctx, "window/showMessage", &lsp.ShowMessageParams{
    	Type: lsp.Warning,
    	Message: fmt.Sprintf("Ignoring new workspace folder %s: %s."+
    	" This is most likely bug, please report it.", added.URI, err),
    	})
    	continue

  • terraform-ls/internal/langserver/handlers/did_change.go

    Lines 31 to 37 in 99e30e1

    	// Versions don't have to be consecutive, but they must be increasing
    	if newVersion <= doc.Version {
    	svc.stateStore.DocumentStore.CloseDocument(dh)
    	return fmt.Errorf("Old version (%d) received, current version is %d. "+
    	"Unable to update %s. This is likely a bug, please report it.",
    	newVersion, doc.Version, p.TextDocument.URI)
    	}

  • terraform-ls/internal/langserver/handlers/initialize.go

    Lines 111 to 117 in a82ef73

    	if !clientCaps.Workspace.WorkspaceFolders && len(params.WorkspaceFolders) > 0 {
    	jrpc2.ServerFromContext(ctx).Notify(ctx, "window/showMessage", &lsp.ShowMessageParams{
    	Type: lsp.Warning,
    	Message: "Client sent workspace folders despite not declaring support. " +
    	"Please report this as a bug.",
    	})
    	}

  • This is also practiced by Terraform (core)
    • https://github.com/hashicorp/terraform/blob/d35bc0531255b496beb5d932f185cbcdb2d61a99/internal/terraform/validate_selfref.go#L28
    • https://github.com/hashicorp/terraform/blob/d35bc0531255b496beb5d932f185cbcdb2d61a99/internal/legacy/helper/schema/provider.go#L235-L240
    • https://github.com/hashicorp/terraform/blob/d35bc0531255b496beb5d932f185cbcdb2d61a99/internal/legacy/helper/schema/provisioner.go#L186-L190
• The ast package's IsModuleFilename and IsVarsFilename work with filenames, not whole paths nor URIs and the same goes for the os package generally
• I may not have been clear about the use of path.Base() in my earlier comments. We still want to handle directory changes. I just used the existing document package, which internally uses path.Base()

[radeksimko]

Also after testing it more I just realized I was wrong about the need of checking just Changed - we'd also need to account for deletions and creations of files within existing (tracked) directories - I'll address that.

[radeksimko]

This turns out to be more complex than I initially anticipated, but I believe my last commit finally accounts for the remaining cases (deletions and creations).

@danishprakash let me know if you have any feedback or questions.

[danishprakash]

@radeksimko hey, I have a question about the recent updates:

• We initially decided not to handle created and deleted events with the assumption that didOpen is triggered at least for the former. Were you able to reproduce a scenario wherein this was not the case? What did you test?

[danishprakash]

When can this happen? The user deleted the only file and then subsequently the directory, too? If yes, then those would also be different events here and that's probably why you're just ensuring the directory is still present or not. Is my understanding correct?

[radeksimko]

This accounts for the deletion of a whole directory.

If yes, then those would also be different events here

Frankly, I don't know, since LSP doesn't go into detail about how clients should or shouldn't group file events. In fact when I tried to rm -rf submodule in VS Code where submodule had some *.tf files, no events were fired at all - which may be a bug in the VS Code language client - unsure.

So in the face of uncertainty I'm just being a little more defensive here.

[danishprakash]

Got it, thanks for the explanation and yes, it's still a little nebulous when it comes to understanding client-side event emits.

may be a bug in the VS Code language client

Do you mean https://github.com/Microsoft/vscode-languageserver-node?

[radeksimko]

It could be either VS Code itself or the language client you linked and it may also be OS-specific problem since file watching is highly dependent on the OS and implementations/behaviours can differ between systems - it would take some debugging to find this out.

[danishprakash]

Curious as to why this check is here since we're doing this earlier:

parentDir := filepath.Dir(rawPath)

[radeksimko]

Right, this should basically never happen - I was just trying to be a bit more defensive here too.

The only realistic scenarios which come to mind involve cases where it's not a file nor a directory, but e.g. symlink or any special type of file - which is mostly relevant on Unix-based systems:
https://pkg.go.dev/io/fs#FileMode

[danishprakash]

Okay, so just yesterday, I got bit by a severe production bug wherein I was copying files from one directory to another. Without a safety check of whether the file is a regular file or not, we ended up doing a cp on a file which was a UNIX pipe and the application just froze indefinitely waiting for the copy to finish which would never be the case. I can totally understand the need for this check here howsoever obvious it might seem.

[radeksimko]

We initially decided not to handle created and deleted events with the assumption that didOpen is triggered at least for the former. Were you able to reproduce a scenario wherein this was not the case? What did you test?

So didOpen is triggered for any open files. What I didn't realize initially was that we actually use the parsed AST and other related metadata of unopened modules e.g. for completion within module block, which will be even more important when #724 is addressed.

[jpogran]

I'm still nervous about opening so many message windows that they can't silence, for things the user can't do anything about, but this all works for the cases I could think of so I don't have any objections to merging this. Thank you both for this work!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.