Race-condition when renaming a file after upload

[Sune1337]

I'm not sure if this is a issue in FluentFTP or the FTP-server i'm using.

I'm uploading a file to a temporary filename to rename the file after the upload is complete.
This is because the folder i'm uploading to is periodically scanned by 3rd party software for certain files, and i need to prevent the 3rd party software to read partially completed uploads.

The following code causes the server to return error "Internal error renaming the file".
I'm guessing the issue is that the client wants to rename the uploaded file before the server thinks it's complete.

// Open a stream to write on the FTP server
using (var ftpStream = ftpClient.OpenWrite("/file.tmp"))
{
    var bytes = encoding.GetBytes(fileContents);
    ftpStream.Write(bytes, 0, bytes.Length);
}

// Move file from temporary upload path to final.
ftpClient.Rename("/file.tmp", "/file.txt");

server log:

(000142)2016-11-04 09:37:20 - ****** (a.b.c.d)> STOR file.tmp
(000142)2016-11-04 09:37:20 - ****** (a.b.c.d)> 150 Opening data channel for file upload to server of "/file.tmp"
(000142)2016-11-04 09:37:20 - ****** (a.b.c.d)> SSL connection for data connection established
(000141)2016-11-04 09:37:20 - ****** (a.b.c.d)> RNFR /file.tmp
(000141)2016-11-04 09:37:20 - ****** (a.b.c.d)> 350 File exists, ready for destination name.
(000141)2016-11-04 09:37:20 - ****** (a.b.c.d)> RNTO /file.txt
(000141)2016-11-04 09:37:20 - ****** (a.b.c.d)> 450 Internal error renaming the file
(000141)2016-11-04 09:37:20 - ****** (a.b.c.d)> QUIT
(000141)2016-11-04 09:37:20 - ****** (a.b.c.d)> 221 Goodbye
(000141)2016-11-04 09:37:20 - ****** (a.b.c.d)> disconnected.
(000141)2016-11-04 09:37:20 - ****** (a.b.c.d)> disconnected.
(000142)2016-11-04 09:37:20 - ****** (a.b.c.d)> 226 Successfully transferred "/file.tmp"
(000142)2016-11-04 09:37:20 - ****** (a.b.c.d)> QUIT
(000142)2016-11-04 09:37:20 - ****** (a.b.c.d)> 221 Goodbye
(000142)2016-11-04 09:37:20 - ****** (a.b.c.d)> disconnected.
(000142)2016-11-04 09:37:20 - ****** (a.b.c.d)> disconnected.

I can work around the problem by manually closing the control-connection for the file transfer.

...
    ftpStream.Write(bytes, 0, bytes.Length);

    // Close control-connection manually
    (ftpStream as FtpDataStream)?.Close();
...

server log:

(000144)2016-11-04 09:47:16 - ****** (a.b.c.d)> STOR file.tmp
(000144)2016-11-04 09:47:16 - ****** (a.b.c.d)> 150 Opening data channel for file upload to server of "/file.tmp"
(000144)2016-11-04 09:47:16 - ****** (a.b.c.d)> SSL connection for data connection established
(000144)2016-11-04 09:47:16 - ****** (a.b.c.d)> 226 Successfully transferred "/file.tmp"
(000144)2016-11-04 09:47:16 - ****** (a.b.c.d)> QUIT
(000144)2016-11-04 09:47:16 - ****** (a.b.c.d)> 221 Goodbye
(000144)2016-11-04 09:47:16 - ****** (a.b.c.d)> disconnected.
(000144)2016-11-04 09:47:16 - ****** (a.b.c.d)> disconnected.
(000143)2016-11-04 09:47:16 - ****** (a.b.c.d)> RNFR /file.tmp
(000143)2016-11-04 09:47:16 - ****** (a.b.c.d)> 350 File exists, ready for destination name.
(000143)2016-11-04 09:47:16 - ****** (a.b.c.d)> RNTO /file.txt
(000143)2016-11-04 09:47:16 - ****** (a.b.c.d)> 250 file renamed successfully
(000143)2016-11-04 09:47:16 - ****** (a.b.c.d)> QUIT
(000143)2016-11-04 09:47:16 - ****** (a.b.c.d)> 221 Goodbye
(000143)2016-11-04 09:47:16 - ****** (a.b.c.d)> disconnected.
(000143)2016-11-04 09:47:16 - ****** (a.b.c.d)> disconnected.

[robinrodricks]

This is an issue in FluentFTP. I'm working on this. The error is because the data stream used for writing the file is parallel to the FTP control stream. I need to find a way to delay the control stream so race conditions don't occur. I'll be adding the alt version in another API, probably called UploadFile() or similar. I'll let you know when its ready.

[robinrodricks]

I have fixed this in 16.0.17 if you use the high level UploadFile and DownloadFile API. This is the code I used to test:

static void TestUploadDownloadFile() {

	using (FtpClient cl = new FtpClient()) {
		cl.Host = m_host;
		cl.Credentials = new NetworkCredential(m_user, m_pass);

		// 10 M file
		cl.UploadFile(@"D:\bigfile.exe", "/public_html/temp/big.txt");
		cl.Rename("/public_html/temp/big.txt", "/public_html/temp/big2.txt");
		cl.DownloadFile(@"D:\bigfile_2.exe", "/public_html/temp/big2.txt");

	}
}

Please check with the latest version and see if it solves your problem.

[tobiasps]

Hi @hgupta9,
I'm using the low-level commands for uploading to be able to get progress info, ie. OpenAppend/OpenWrite and stream methods Write, Flush, then Close and Dispose.
After completed upload I'm calling Rename on the client, however what I'm seeing is that theres is a long wait before the Rename completes. Also if I do not insert a thread delay I get an exception on the rename command. Furthermore the bigger the file the longer the delay.
I am really impressed by the transfer speed, but unfortunately it takes longer renaming the file than transferring it. I'm testing with files larger than 100MB and I'm using the FluentFTP in a Xamarin project for iOS and Android.
I upload to a temporary name and then rename as I have a backend software that watches the upload folder.

Can I do anything to shorten the delay?

I have also tried disconnecting and reconnecting, in this case the disconnect is delayed. Once reconnected the rename is instantaneous.

[robinrodricks]

Can you please add a separate issue for this as I see no connection with the OP's issue. And if you can use my code as a reference instead of coding from scratch; basically I upload in chunks, and various other stuff that might be useful and/or might solve your problem automatically. Please see UploadFileInternal