"File does not exist" every time when using OpenFile

[ericcancil]

As i said in the title, I get "File does not exist" every time when using OpenFile. I can successfully call Open on the same file without any error, and am wondering what is causing this issue.

Thank you

[eikenb]

Both OpenFile() and Open() use the exact same underlying call. The only difference is that Open() has the readonly flag hardcoded in where OpenFile() requires you pass the flags. So my best guess with the information available is that you are calling it with a bad flag. You should be passing in flags from the os package. Eg. client.OpenFile("/file/path", os.O_RDONLY) should behave the same as client.Open("/file/path").

[eikenb]

@ericcancil Just pinging you in case you didn't see my previous post. I'm not sure if you get emailed notifications of my reply w/ vs w/o mentioning you.

[ericcancil]

@eikenb
I'm sorry, these were going to spam for some reason. Thanks for your reply.

I'm writing the file as follows

func checkForLogFile(client *sftp.Client) {
	if _, err := client.Stat(logFile); os.IsNotExist(err) {
		var file, createErr = client.Create(logFile)
		if createErr != nil {
			panic(createErr)
		} else {
			log.Println("log file created!")
			defer file.Close()
		}
	} else {
		if err != nil {
			panic(err)
		} else {
			log.Println("Log file already exists")
		}
	}
}

Then in the future I am trying to write to it as such

func writeLogFile(fileInfo FileAndPath, log FileLog) error {
	if f, err := fileInfo.Client.OpenFile(logFile, os.O_RDWR|os.O_CREATE|os.O_TRUNC); err != nil {
		return err
	} else {
		defer f.Close()

		if _, err := f.Write([]byte(log.String())); err != nil {
			return err
		}
	}
	return nil
}

I'm using os.O_RDWR|os.O_CREATE|os.O_TRUNC as that is what the create call uses. Yet I am still getting file does not exist every time.

Any ideas?

[eikenb]

Which SFTP server is your client connecting to? Testing your code with my local OpenSSH/SFTP server and it works.

[ericcancil]

When I test locally with docker it works as well. When i connect to any remote sftp server
(which i have no insight into the setup) I am getting these issues.

You don't see anything funny as far as my code? Thanks for answering back so quickly. Really appreciate it.

[eikenb]

Your code looks fine and should work. Does the checkForLogFile() call return the error as well, or just the writeLogFile() call?

Have you tried different flags, maybe leave off os.O_TRUNC? My guess is the server doesn't handle the flags correctly.

You could also try hitting it with this command, if available, it might tell you which server it is.

nmap -A -Pn -T4 -p 22 [host]

[ericcancil]

I tried with just os.O_TRUNC with same results

Here are the results of nmap

Starting Nmap 7.12 ( https://nmap.org ) at 2018-01-02 23:54 GMT
Nmap scan report for ftp.med-metrix.com (209.255.119.163)
Host is up (0.020s latency).
rDNS record for 209.255.119.163: 209-255-119-163.ip.mcleodusa.net
PORT   STATE SERVICE VERSION
22/tcp open  ssh     (protocol 2.0)
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service :
SF-Port22-TCP:V=7.12%I=7%D=1/2%Time=5A4C1BBA%P=x86_64-openwrt-linux-gnu%r(
SF:NULL,21,"SSH-2\.0-1\.82_sshlixx\x20Med-Metrix\r\n");
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: game console
Running (JUST GUESSING): Microsoft embedded (85%)
Aggressive OS guesses: Microsoft Xbox game console (modified, running XboxMediaCenter) (85%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 2 hops

TRACEROUTE (using port 22/tcp)
HOP RTT      ADDRESS
1   0.05 ms  ip-172-17-0-1.ec2.internal (172.17.0.1)
2   28.51 ms 209-255-119-163.ip.mcleodusa.net (209.255.119.163)

OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 11.84 seconds

[ericcancil]

If i send os.O_RDONLY it works, even tho I'm writing it as os.O_RDWR|os.O_CREATE|os.O_TRUNC

[eikenb]

If that nmap system detect is right, the it doesn't look like I'll be able to replicate this. Are you able to upload files using openssh's sftp client?

It sounds like you have a workaround. That if you open the file with os.O_RDONLY it gives you the filehandle and lets you write to it?

[ericcancil]

writing fails with a does not exist error. Very strange.

[eikenb]

writing fails with a does not exist error. Very strange.

When using openssh's sftp client?

[ericcancil]

Works when using all clients except the library

[eikenb]

Can you tell me anything more about the server?

I'll have to look at what the client lib does in comparison to openssh's version. Look for differences. But it would be a lot easier if I could reproduce the issue somehow.

[eikenb]

One difference is that openssh's function to upload a file sends an open packet with the flags;

SSH2_FXF_WRITE | SSH2_FXF_CREAT | SSH2_FXF_TRUNC

Where our client always sends;

ssh_FXF_READ | ssh_FXF_WRITE | ssh_FXF_CREAT | ssh_FXF_TRUNC

You could replicate what openssh does by calling OpenFile() like so...

Client.OpenFile(logFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC)

[ericcancil]

unfortunately the same thing. Going to try and skip this feature right now as we are backing things up in s3 and we have versioning turned on. THanks a lot for your help

[eikenb]

If there is anything more you can tell me about the server it might be helpful. I'll spend a bit more time looking over what openssh does to see if any more differences jump out at me.

[eikenb]

You could try specifying a MaxPacket size to something smaller than default. I found this in the openssh code...

/* Some filexfer v.0 servers don't support large packets */
if (ret->version == 0)
	ret->transfer_buflen = MIN(ret->transfer_buflen, 20480);

To set this using our code...

client, err := sftp.NewClient(sshConn, sftp.MaxPacket(20480))

The normal default is 32768.

[by-cx]

@eikenb Thank for the "SSH2_FXF_WRITE | SSH2_FXF_CREAT | SSH2_FXF_TRUNC" flags for OpenFile, it solved my problem with one strange SFTP server.

[ilovelinux]

I get the same error. I tried both #212 (comment) and #212 (comment) your solutions but none of them worked.
Here's my OpenSSH version:

$ ssh -V
OpenSSH_7.8p1, OpenSSL 1.1.1  11 Sep 2018

========== UPDATE ==========
I don't know if it can help but as you can see I'm trying to create file using the full path as path variable.

Client.Create("\path\of\file")
Client.OpenFile("\path\of\file", os.O_WRONLY|os.O_CREATE|os.O_TRUNC)
Client.OpenFile("\path\of\file", os.O_WRONLY|os.O_CREATE)

[eikenb]

@ilovelinux What server are you using? Openssh?

[ilovelinux]

Yes, the version I sent is the server's OpenSSH version. Client runs in a clear installation of Windows XP

[eikenb]

Can you provide a small program that re-creates the problem? I test against OpenSSH manually fairly often and the client integration tests also test against OpenSSH. So I'm not sure if I'll be able to reproduce it, but with a self-contained example maybe we can find help from someone with Windows 10 access to verify the issue. Thanks.

[ilovelinux]

Gosh! I found the bug! It was my fault!
My program didn't correctly check if the remote folder to use as destination for files exist (to create it if it doesn't exist) so basically I was trying to transfer files in an non-existent directory!
Everything works as expected, it's a great library 😄

[RandyField]

OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017