ssh deployer errors with "unknown keys [location_file]"

[aranw]

I've tried both setting up my own test weaver application and using the collatz example and both times when running

❯ weaver ssh deploy weaver.toml

I get the following error message

unable to parse ssh config: section "ssh" has unknown keys [locations_file]

I've tried looking at the code but nothing seems to stand out to me at a quick glance as to why this is erroring. I don't fully understand how all the code works just yet but thought I'd raise the issue and continue debugging

[dnephin]

Nice bug report! I'm just learning the code as well, but I may have an idea about this problem.

weaver/internal/tool/ssh/deploy.go

Lines 181 to 186 in 35727ae

	type sshConfigSchema struct {
	LocationsFile string `serverweaver_metric:"locations_file"`
	}
	parsed := &sshConfigSchema{}
	if err := runtime.ParseConfigSection(sshKey, shortSSHKey, app, parsed); err != nil {
	return nil, fmt.Errorf("unable to parse ssh config: %w", err)

I believe the serverweaver_metric struct tag on line 182 is wrong. It doesn't appear anywhere else in the code, and ParseConfigSection (which receives that struct) seems to use https://pkg.go.dev/github.com/burntsushi/toml#Decode to decode the config. That function expects a toml struct tag.

That leaves the locations_file field from the config as "undecoded", which causes the error here:

weaver/runtime/config.go

Lines 87 to 88 in 35727ae

	if unknown := md.Undecoded(); len(unknown) != 0 {
	return fmt.Errorf("section %q has unknown keys %v", key, unknown)

I suspect this diff should fix it, but I'm not familiar enough with the testing of this package to add test coverage for the change:

diff --git a/internal/tool/ssh/deploy.go b/internal/tool/ssh/deploy.go
index 6a6e276..9be6534 100644
--- a/internal/tool/ssh/deploy.go
+++ b/internal/tool/ssh/deploy.go
@@ -179,7 +179,7 @@ func getLocations(app *protos.AppConfig) ([]string, error) {
 	const shortSSHKey = "ssh"
 
 	type sshConfigSchema struct {
-		LocationsFile string `serverweaver_metric:"locations_file"`
+		LocationsFile string `toml:"locations_file"`
 	}
 	parsed := &sshConfigSchema{}
 	if err := runtime.ParseConfigSection(sshKey, shortSSHKey, app, parsed); err != nil {

[rgrandl]

Hi @aranw. Thanks for pointing out the issue. @dnephin this is one of the problem, great find :). There is another small issue we introduced while renaming the project. I am creating a fix.

[aranw]

Ah thanks for finding this so quickly. I didn't get too deep into identifying the issue before having to go out. I'll try this diff either later tonight or tomorrow at some point

[rgrandl]

Sorry for the inconvenience. Please try the diff and let us know if you have any issues running the SSH deployer.

[aranw]

Yeah that looks like it has progressed a little further now onto a new error relating to creating deployment directory

Thanks for the quick turn around with a fix!

[rgrandl]

Hmm, can you share the error? I tried on my local machine and it works fine. Are you testing on your local machine or on multiple machines?

[aranw]

It is a remote server

unable to create deployment directory at location #####: exit status 1

##### is the name of my host alias in my ssh config file

[rgrandl]

Got it. We create a deployment directory on each remote machine where Service Weaver will be deployed. Right now we try to create the directory using ssh and the location is os.TempDir()/deploymentId. https://github.com/ServiceWeaver/weaver/blob/main/internal/tool/ssh/deploy.go#L147

Can you try if simply creating the remote directory over SSH works? e.g., ssh #### "mkdir -p /tmp/bla". I am wondering whether you have some permission issues. Are you running linux machines?

[aranw]

@rgrandl yeah running that command it worked fine

[aranw]

I added some output to the command to see what the tool was doing and this seems to be the error

mkdir: cannot create directory ‘/var/folders’: Permission denied

Not sure where it is getting /var/folders yet need to trace the code a bit more

[aranw]

This is the command it is running

I'm guessing this is a folder on my local machine

/usr/bin/ssh -v ### mkdir -p /var/folders/sq/3sdsh9n14p790p7xwhdlvdzc0000gn/T/915574ab-82d5-47f1-8efa-47f5062f03cf

[rgrandl]

Hmm, this is interesting. On linux, the temp dir is /tmp. What is your setup and which OS are you running? It feels like macOS?

It seems like your ssh client doesn't have privileges to create a directory in /var/folders, maybe? Alternatively you can try to change the os.tempDir() to a custom directory where you can do writes over SSH and check if it works for now.

I have a macOS laptop and I will give it a try as well. To be fair, we tested the SSH deployer only on linux machines.

[aranw]

Ahh so I understand what is happening now

weaver/internal/tool/ssh/deploy.go

Line 143 in 35727ae

remoteDepDir := filepath.Join(os.TempDir(), dep.Id)

On this line you call os.TempDir() but because I am running this from my laptop (macOS) that call to os.TempDir() returns a temp directory path for my local machine not for the remote ssh host that I am connecting too

Edit: I can create a separate issue for this if you want?

[rgrandl]

That's a good point. Yeah, we didn't try to deploy it using SSH from our laptop on a cluster. Thanks for the finding. Sure, send us a fix. Thanks for fixing this :).

[aranw]

@rgrandl I'll write up a new issue and little summary of the problem later today. I did some digging into the issue last night but was late for me (GMT timezone) so ended up not working on a fix. I have an idea for a possible fix but will discuss in more detail a possible solution in the issue later

[rgrandl]

@aranw thanks a lot for your help. Sounds good.