Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: os: Create/Open/OpenFile() set FILE_SHARE_DELETE on windows #32088

Closed
networkimprov opened this issue May 16, 2019 · 193 comments

Comments

@networkimprov
Copy link

commented May 16, 2019

On Linux & MacOS we can write this; on Windows it fails with a "sharing violation":

path := "delete-after-open"
fd, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0600)
if err != nil { ... }
err = os.Remove(path)            // or os.Rename(path, path+"2")
if err != nil { ... }
fd.Close()

If you develop on Windows and deploy to Linux etc, and your code relies on this undocumented GOOS=windows behavior of os.Rename() & .Remove(), it is broken and perhaps vulnerable. Note that package "os" has fifteen mentions of other Windows-specific behavior.

To fix this, syscall.Open() at https://golang.org/src/syscall/syscall_windows.go#L272
needs sharemode |= FILE_SHARE_DELETE

Microsoft recommends this be made the default: #32088 (comment)
Rust made it a default: https://doc.rust-lang.org/std/os/windows/fs/trait.OpenOptionsExt.html
Mingw-w64 made it a default seven years ago:
https://sourceforge.net/p/mingw-w64/code/HEAD/tree/stable/v3.x/mingw-w64-headers/include/ntdef.h#l858
Erlang made it a default over six years ago: erlang/otp@0e02f48
Python couldn't adopt it due to limitations of MSVC runtime: https://bugs.python.org/issue15244

Therefore syscall.Open() should use file_share_delete by default, and syscall should provide both:
a) a global switch to disable it (for any existing apps that rely on its absence), and
b) a flag for use with os.OpenFile() to disable it on a specific file handle.

Update after #32088 (comment) by @rsc:
a) os.Create/Open/OpenFile() should always enable file_share_delete on Windows,
b) syscall.Open() on Windows should accept a flag which enables file_share_delete, and
c) syscall on Windows should export a constant for the new flag.

The docs for os.Remove() should also note that to reuse a filename after deleting an open file on Windows, one must do: os.Rename(path, unique_path); os.Remove(unique_path).

Win API docs
https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-deletefilea
https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-createfilea

If there is a reason not to do this, it should be documented in os.Remove() & .Rename().

cc @alexbrainman
@gopherbot add OS-Windows

@bradfitz bradfitz added this to the Go1.14 milestone May 16, 2019
@bradfitz

This comment has been minimized.

Copy link
Member

commented May 16, 2019

@alexbrainman

This comment has been minimized.

Copy link
Member

commented May 18, 2019

syscall.Open() at https://golang.org/src/syscall/syscall_windows.go#L272
should use sharemode := ... | FILE_SHARE_DELETE

Why should it?

Alex

@networkimprov

This comment has been minimized.

Copy link
Author

commented May 18, 2019

FILE_SHARE_DELETE enables the code example above, which works on MacOS & Linux but fails on Windows. It's also necessary for:

fd, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0600)
defer fd.Close()
_, err = fd.Write(...)
err = fd.Sync()  // file now safe to share
err = os.Rename(path, path+"2")
_, err = fd.Read(...)
@alexbrainman

This comment has been minimized.

Copy link
Member

commented May 19, 2019

FILE_SHARE_DELETE enables the code example above, which works on MacOS & Linux but fails on Windows.

Why don't we adjust MacOS & Linux code instead?

It's also necessary for:

I don't understand what you are trying to say.

Alex

@mattn

This comment has been minimized.

Copy link
Member

commented May 19, 2019

@networkimprov You must call Remove after Close() on Windows.

path := "delete-after-open"
fd, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0600)
if err != nil { panic(err) }
fd.Close()
err = os.Remove(path)            // or os.Rename(path, path+"2")
if err != nil { panic(err) }
@networkimprov

This comment has been minimized.

Copy link
Author

commented May 19, 2019

@alexbrainman when doing reliable file I/O (as for databases), it's standard practice to create a file with a temporary name, write it, fsync it, and rename it.

Open files can be renamed or deleted by default on Unix. It seems like an oversight that the Windows flag for this capability is not set. I doubt we'll convince the Go team to change the way it works for Linux & MacOS :-)

@mattn pls apply the fix I described and try the code I posted.

@alexbrainman

This comment has been minimized.

Copy link
Member

commented May 19, 2019

I doubt we'll convince the Go team to change the way it works for Linux & MacOS :-)

I am fine the way things are now.

Alex

@networkimprov

This comment has been minimized.

Copy link
Author

commented May 19, 2019

Is there a rationale for omitting this common capability in Windows?

Can you provide a switch in syscall_windows.go so that we can select the Unix behavior at program start?

@mattn

This comment has been minimized.

Copy link
Member

commented May 19, 2019

I'm okay that we add new API or flags. But I have objection to change current behavior. Since FILE_SHARE_DELETE is not same as Unix behavior.

#include <windows.h>
#include <stdio.h>

int
main(int argc, char* argv[]) {
  HANDLE h = CreateFile("test.txt",
      GENERIC_READ | GENERIC_WRITE,
      FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
      NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
  getchar();
  char buf[256] = {0};
  DWORD nread;
  printf("%d\n", ReadFile(h, buf, 256, &nread, NULL));
  printf("%s,%d\n", buf, nread);
  return 0;
}

Comple this code on Windows, and try to run as test.exe. While this app is waiting hit-a-key, open new cmd.exe, delete "test.txt" like following.

image

The file can be deleted but remaining there while the process exists. So this change will not work well for your expected.

@networkimprov

This comment has been minimized.

Copy link
Author

commented May 19, 2019

I realize the directory entry isn't removed but the docs say

Subsequent calls to CreateFile to open the file fail with ERROR_ACCESS_DENIED.

So I don't understand your screen log.

Anyway, a switch like this would be fine:

syscall.OpenFileShareDelete = true
@ericlagergren

This comment has been minimized.

Copy link
Contributor

commented May 19, 2019

I’d like to mention that I’ve been bitten by this at work before.

Basically we had:

f, err := os.Open(...)
if err != nil { ... }
defer f.Close()

// lots of code

if err := os.Rename(...); err != nil { ... }

The code ran fine on our unix-based CI platforms, but exploded on Windows.

Assuming there aren’t any weird side effects, it would be nice if things just worked.

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 19, 2019

IIRC, we've made a few adjustments to GOOS=windows & plan9 behavior in the past to more closely match Unix semantics. I wouldn't mind making this be another such case if the semantics are close enough. @mattn's comment, however, suggests the behavior is not close enough so it might not be worth it.

I don't want to see some global option, though. That just seems like a debugging nightmare.

@guybrand

This comment has been minimized.

Copy link

commented May 20, 2019

@bradfitz
It would probably not, please refer to my comment here
https://groups.google.com/forum/#!topic/golang-dev/R79TJAzsBfM
or if you prefer I can copy the content here, as there is also a possible solution, although I would not implement it as the default behavior, as this would not be consistent with how windows programs behave but rather a workaround to create a similar cross-os experience.

@networkimprov

This comment has been minimized.

Copy link
Author

commented May 20, 2019

@guybrand writes in the golang-dev thread:

my program is writing a file called "my.data", and then calls another program and does not wait for it to end ... this program lets say uploads this file which takes 10 seconds (lets call this other program "uploader") .
...
When my program calls .Remove on Windows (FILE_SHARE_DELETE is on):
...
uploader would receive an error telling it the file is removed (probably EOF).

Assuming "uploader" opens the file before "my program" calls os.Remove(), I think you contradicted the Win API docs:

DeleteFile marks a file for deletion on close. Therefore, the file deletion does not occur until the last handle to the file is closed. Subsequent calls to CreateFile to open the file fail with ERROR_ACCESS_DENIED.

Re "pairing an _open_osfhandle() of the CreateFile to an _fdopen", can you point to example code?

@mattn

This comment has been minimized.

Copy link
Member

commented May 20, 2019

If we add FILE_SHARE_DELETE, many programmer will mistakenly use it to simulate Unix behavior. In this case, you can make a function separated by build-constraints for each OSs. And it should return os.NewFile(), use FILE_SHARE_DELETE on Windows.

@networkimprov

This comment has been minimized.

Copy link
Author

commented May 20, 2019

make a function separated by build-constraints for each OSs... return os.NewFile(), use FILE_SHARE_DELETE on Windows

To retain the functionality of os.OpenFile() this plan appears to require reimplementing all of:

os.OpenFile()
openFileNolog()
openFile()
openDir()
newFile()
fixLongPath()
syscall.Open()
makeInheritSa()

@guybrand

This comment has been minimized.

Copy link

commented May 20, 2019

@networkimprov

@guybrand writes in the golang-dev thread:

my program is writing a file called "my.data", and then calls another program and does not wait for it to end ... this program lets say uploads this file which takes 10 seconds (lets call this other program "uploader") .
...
When my program calls .Remove on Windows (FILE_SHARE_DELETE is on):
...
uploader would receive an error telling it the file is removed (probably EOF).

Assuming "uploader" opens the file before "my program" calls os.Remove(), haven't you contradicted the Win API docs?

DeleteFile marks a file for deletion on close. Therefore, the file deletion does not occur until the last handle to the file is closed. Subsequent calls to CreateFile to open the file fail with ERROR_ACCESS_DENIED.

Re "pairing an _open_osfhandle() of the CreateFile to an _fdopen", can you point to example code?

I dont see a contradiction with WIndows API, please point what looks like a contridiction.

As for a code sample, I Googled a bit, and found this:
http://blog.httrack.com/blog/2013/10/05/creating-deletable-and-movable-files-on-windows/

BUT
Please note - this will only give you a nix-like behavior internally, any other external program working with it will break it (as it 99% using standard windows API

@guybrand

This comment has been minimized.

Copy link

commented May 20, 2019

@networkimprov
If you mean "this sounds like "dir my.data" would display the file, as far as I remember this was the behavior until windows .... XP or 7 (dont remember which) and changed since (perhaps the explorer just hides it somehow) - I can retest the next time I launch my windows env. (happens every few week...)

If you mean "this sounds like other processes with a handle to the file would still be able to read and write to the file", I would bet lunch the second you read write to such a file you do get an "EOF" style error - but again need to confirm to be 100% positive.

In either case, my point would be (at this point - I am taking "sides" in the "native like" vs " os-agnostic" point) - even if you implement _fdopen style solution, you would get inconsistency between your service and all other executables you collaborate with, so the use could only be in interaction with other go executables (or rare services that DO use fd's directly).

In other words - your app would be the "smartest kid in class" - that no other kid can understand.
Two examples from many I can think of:
Inputs:
My app downloads a file,the antivirus identifies its harfull and deletes/quarantines (==sort of rename) it, if I use fd - my app would still be able to do whatever it want with it (which is coll, but may end up hitting a virus...)
outputs:
my app pipes a file to another ("uploader" like) service, and deletes it, I even bothered and wrote a tester in go to see that all is working fine - and the test passes.
Now instead of my go test, I use filezilla, we transfter, dropbx API, whatever
it would fail/not behave in the same manner my test works...

Do you still think changing this to the default behavior makes sense ?

@alexbrainman

This comment has been minimized.

Copy link
Member

commented May 20, 2019

Is there a rationale for omitting this common capability in Windows?

I have never considered that question. I do not know.

The way Go files work on Windows is consistent with all other developers tools I have used in my life. It would be surprising to me, if Go files would work as you propose. I also suspect, it would break many existing programs.

Alex

@networkimprov

This comment has been minimized.

Copy link
Author

commented May 20, 2019

I also suspect, it would break many existing programs.

@alexbrainman I also suggested a switch to enable it, instead of changing the default.

@bradfitz there is syscall.SocketDisableIPv6, that's not really different from a flag to adjust syscall.Open() behavior.

@guybrand

This comment has been minimized.

Copy link

commented May 20, 2019

Since syscall_windows*.go states
func Open(path string, mode int, perm uint32) (fd Handle, err error) {
....
sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE)

and type_windows*.go has
FILE_SHARE_DELETE = 0x00000004

All is quite ready, the only question is how to implement the flag.
I do not like the global option as well as its not explicit, and while a single developer may remember he has set os.DeleteUponLastHandleClosed = 1, its not a good practice for long term or multiple developer.
Other options would either be setting a specific reserved number for flags, like in:
fd, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.DELETE_WHEN_FREED, 0600)
whereas DELETE_WHEN_FREED can even be 0 for env. other than windows,

Another option would be to use the perm parameter, which is not supported for windows, this may get a little awkward
fd, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 777)
777 is reserved, so we would either need a 1777 or -777 ro support both systems
to make is readable DELETE_WHEN_FREED | 777

last option I can think of is os.OpenDeletableFile(
Which would os.OpenFile on nix's and
turn
sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE)
on windows

all the above solutions are simple to implement, a little more time to test (cross os), just need a voter...

@havoc-io

This comment has been minimized.

Copy link

commented May 20, 2019

One way to support this behavior without changing the defaults or extending the API surface might be to just accept syscall.FILE_SHARE_DELETE in the flag parameter of os.OpenFile on Windows and fold it into the computed sharemode value. Since the syscall.O_* (and hence os.O_*) flags on Windows use made up values anyway, they could be engineered to avoid colliding with any Windows-specific flags that one wanted to include. Fortunately, they already avoid collisions with syscall.FILE_SHARE_DELETE.

In any event, I would strongly oppose changing the default behavior. Making FILE_SHARE_DELETE the default would put you in the same position as POSIX systems in terms of being unable to ensure race-free filesystem traversal. Having this flag be optional is why Windows doesn't need the equivalent of openat, renameat, readlinkat, etc.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented May 21, 2019

I haven't really thought about what we should do here, but I want to make one thing clear:

Can you provide a switch in syscall_windows.go so that we can select the Unix behavior at program start?

We will not be doing this. That would make it impossible for a single program to use different packages that expect different behavior.

@guybrand

This comment has been minimized.

Copy link

commented May 21, 2019

@havoc-io
Your suggestion would create error prone programs as syscall.FILE_SHARE_DELETE differs from POSIX std behavior.
Please note the authors of syscall_windows*.go were aware of the FILE_SHARE_DELETE flag (its in there) and decided using it would not be the preferred behavior.
Why?

lets take Liam's code, he
defer fd.Close()
deletes/renames
and then read/writes

so actual sort order is
open
mark as deleted
read/write (and probably cross process/cross go routine read write as well)
close

Current windows behavior alerts the developer "this is wrong", developer would probably push the delete to be defer'ed as well
If you Add FILE_SHARE_DELETE, windows would allow you to "mark as delete even though the file open" BUT another process/goroutine running concurrently that would try to access the file between the delete and the close (which is probably the longer task in this code) would fail
Result: instead of a consistent "you can do that" behavior - you would get a "sometimes - especially when there are many concurrent users it fails and I dont know why.
So your not solving the problem, just handling a specific use case on which the developer "only run that once"
My vote is for the consistent behavior of course...

How is that diff from Posix?
Once marked as deleted the file is no longer on the file table, it only exists as an fd, allowing another routine/process to create a file with the very same name.

I think we should stop looking for a "same solution" as there are differences we will not resolve within go (case sensitive filenames ? :) we will let Linux 5.2 do that...)

Altogether it looks like looking for a solution for a temporary file, if it is, windows supports GetTempFileName which can be considered as a standard solution for a file that is "used and then trashed" .

If on the other hand we wish to allow a developer to defer deletes in windows, that's possible, see my suggestions above, but the developer must take responsibility for that, and understand the consciousness - therefore must be a flag with a good name convention.

@networkimprov

This comment has been minimized.

Copy link
Author

commented May 21, 2019

The primary use for this feature is to rename an open file after creating it:

fd, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0600)
_, err = fd.Write(...)
err = fd.Sync()  // file now safe to share
err = os.Rename(path, shared_path)
// os.Close() at program exit

I don't know why you wouldn't add a flag os.O_WRNDEL (windows rename/delete; a no-op on other platforms), and document that mode's differences with Unix; that a deleted file remains in its directory, but can't be opened, until os.Close(). Also mention that moving the file to a temp dir before deleting it provides more Unix-like behavior.

@havoc-io

This comment has been minimized.

Copy link

commented May 21, 2019

@guybrand I think perhaps you're misunderstanding my proposal. I'm not advocating that FILE_SHARE_DELETE be enabled by default - in fact I'm opposing it for the reasons mentioned in the second paragraph of my comment. My proposal was for a mechanism that allows users to opt-in to FILE_SHARE_DELETE behavior (on a per-file basis) while still using the os package's file infrastructure. This proposal provides a mechanism for doing so without extending the API surface of any package.

I don't know why you wouldn't add a flag os.O_WRNDEL (windows rename/delete; a no-op on other platforms)

@networkimprov That's essentially what I'm proposing, except that there's no point in defining a new flag since a perfectly valid one already exists: syscall.FILE_SHARE_DELETE. The only implementation change would be to watch for this flag in syscall.Open on Windows and add checks to ensure that no future additions to syscall.O_*/os.O_* flags collide with this flag. The documentation for os.OpenFile could then be updated to reflect acceptance of this flag on Windows.

@guybrand

This comment has been minimized.

Copy link

commented May 21, 2019

@havoc-io sorry for the misunderstanding, this was my takeout from:
" ... to just accept syscall.FILE_SHARE_DELETE ... into the computed sharemode..."

Adding os.O_WRNDEL matches with my second suggestion apart from not being explicit enough for the developer in terms of "what would be the behavior", perhaps os.WADRNDEL - Windows allow deferred rename/delete) .

craig bot pushed a commit to cockroachdb/cockroach that referenced this issue Sep 29, 2019
41164: storage: fix an error message when trying to backup to local drive on… r=darinpp a=darinpp

… Windows

An often occuring pattern is:
```
{
...
f, _ := os.Create(someFile)
defer f.Close()
...rename or delete the file...
}
```
This does not work on Windows by default. It can be made to work (kindof) when
the file is opened with a special flag but the behaviour is not the same as
on a posix system.
There seems to be an effort to fix this in go as seen here golang/go#32088
Also the default for the newer Windows versions seems to be to allow this golang/go#32088 (comment)
Until then - we should do an extra close before remove or delete (assuming these are the last operations).

Fixes #41085

Release justification: bug fix for existing functionality

Release note: None

Co-authored-by: Darin <darinp@gmail.com>
@networkimprov

This comment has been minimized.

Copy link
Author

commented Sep 29, 2019

Below is a working demo of standard-practice log rotation, with 100 goroutines each logging 50 lines at 0.1-2sec intervals every time they open the log file, and another goroutine rotating the log at 1min intervals. I ran this for several hours yesterday on a Win7 laptop with zero errors.

It's built with a patch to syscall.Open() that enables FILE_SHARE_DELETE; it fails otherwise. One could concoct a more complex logging scheme without this flag, but there are many other uses for it; my own code renames open files for different reasons.

@rsc, I believe this completes the case for your suggestion, if there were still any doubts. (And as noted above, I posted repeatedly in golang-dev and golang-nuts asking for projects that rely on the current behavior, with zero results.)

package main

import (
   "fmt"
   "os"
   "math/rand"
   "syscall"
   "time"
)

const kLogFile = "winrotate"

func main() {
   syscall.Open_FileShareDelete = true
   rand.Seed(time.Now().UnixNano())

   for a := 0; a < 100; a++ {
      go runLog()
   }

   fmt.Println("rotating to "+ kLogFile +"N.txt")
   for a := 0; true; a++ {
      if a >= 5 {
         a = 0
      }
      time.Sleep(60 * time.Second)
      err := os.Rename(kLogFile +".txt", kLogFile + string('0'+a) +".txt")
      if err != nil {
         fmt.Println(err)
         os.Exit(1)
      }
   }
}

const kLineVal = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"

func runLog() {
   aLine := make([]byte, 102)
   for {
      aFd, err := os.OpenFile(kLogFile +".txt", os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0600)
      if err != nil {
         fmt.Println(err)
         return
      }
      for _, aV := range kLineVal {
         for a := range aLine {
            aLine[a] = byte(aV)
         }
         aEnd := aV - ('z' - 100) // limit line len to 100
         copy(aLine[aEnd:], []byte{'\r','\n'})
         _, err = aFd.Write(aLine[:aEnd + 2])
         if err != nil {
            fmt.Println(err)
            return
         }
         time.Sleep(time.Duration(100 + rand.Intn(1900)) * time.Millisecond)
      }
      err = aFd.Close()
      if err != nil {
         fmt.Println(err)
         return
      }
   }
}

Patch to syscall_windows.go:

diff --git a/src/syscall/syscall_windows.go b/src/syscall/syscall_windows.go
index de05840..e1455d5 100644
--- a/src/syscall/syscall_windows.go
+++ b/src/syscall/syscall_windows.go
@@ -245,6 +245,8 @@ func makeInheritSa() *SecurityAttributes {
 	return &sa
 }
 
+var Open_FileShareDelete = false
+
 func Open(path string, mode int, perm uint32) (fd Handle, err error) {
 	if len(path) == 0 {
 		return InvalidHandle, ERROR_FILE_NOT_FOUND
@@ -270,6 +272,9 @@ func Open(path string, mode int, perm uint32) (fd Handle, err error) {
 		access |= FILE_APPEND_DATA
 	}
 	sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE)
+	if Open_FileShareDelete {
+		sharemode |= FILE_SHARE_DELETE
+	}
 	var sa *SecurityAttributes
 	if mode&O_CLOEXEC == 0 {
 		sa = makeInheritSa()
darinpp added a commit to darinpp/cockroach that referenced this issue Sep 30, 2019
… Windows

An often occuring pattern is:
{
...
f, _ := os.Create(someFile)
defer f.Close()
...rename or delete the file...
}

This does not work on Windows by default. It can be made to work (kindof) when
the file is opened with a special flag but the behaviour is not the same as
on a posix system.
There seems to be an effort to fix this is go as seen here
golang/go#32088
Also the default for the newer Windows versions seems to be to allow this
golang/go#32088 (comment)
Until then - we should defer a remove or delete to be done after the Close.

Fixes cockroachdb#41085

Release justification: bug fix for existing functionality

Release note: None
@mattn

This comment has been minimized.

Copy link
Member

commented Oct 2, 2019

@networkimprov The file is not renamed by external process. If renaming the file is doing in same process, there are some other ways to implement it.

https://github.com/mattn/go-sizedwriter/blob/master/_example/example.go#L14

@alexbrainman

This comment has been minimized.

Copy link
Member

commented Oct 2, 2019

Below is a working demo of standard-practice log rotation

Thank you for your example, @networkimprov . I can see now what you are trying to do.

The only issue with your code is that, if some external process will open files you are writing to without FILE_SHARE_DELETE, then you would have the same problem as you have now regardless of changes we make in Go.

@cpuguy83 I won't be looking at your issue (as I promised here #32088 (comment) ), because @networkimprov example give me a picture of what he is trying to do.

Alex

@networkimprov

This comment has been minimized.

Copy link
Author

commented Oct 2, 2019

Alex, one can loop the rename attempt if outside access is allowed, or restrict the active log file if not.

Mattn, the rotate rename is usually done by a separate process.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 2, 2019

It seems like there is no clear consensus here. The developers with Unix backgrounds seem in favor of this, but two of our most active Windows developers (@alexbranman and @mattn) are not. The change would also only truly unify the latest version of Windows that has the new Unix-like behavior for FILE_SHARE_DELETE.

It may be worth revisiting this topic in a few years, to see how widely available the new flag behavior is and whether other languages have converged to a common behavior. If someone wants to file a new issue in, say, two years or so to rethink this, that would be fine.

But for now, given the lack of consensus, this seems like a likely decline.

Leaving open for a week for final comments.

@cpuguy83

This comment has been minimized.

Copy link

commented Oct 2, 2019

Is there contention on just making the option available?

@markdascher

This comment has been minimized.

Copy link

commented Oct 2, 2019

If we're looking for more voices to chime in, I'd definitely be in favor of doing something. Right now, the only solution is to literally copy a bunch of code out of Go's standard library, change one line, and then maintain that fork forever. Every implementation of a log monitor or live "tail" eventually seems to do this.

For an example, see https://github.com/elastic/beats/blob/master/libbeat/common/file/file_windows.go#L85-L103

@guybrand

This comment has been minimized.

Copy link

commented Oct 2, 2019

@thaJeztah

This comment has been minimized.

Copy link
Contributor

commented Oct 2, 2019

Definitely in favour of at least have 1. (Allow the use of the flag FILE_SHARE_DELETE when opening a file)

@havoc-io

This comment has been minimized.

Copy link

commented Oct 2, 2019

Just to revisit my previous proposal and follow-up with example implementation:

The syscall.FILE_SHARE_DELETE flag would fit nicely in the os.OpenFile's flag parameter and would provide a trivial mechanism to enable this behavior on-demand. It doesn't collide with any other os.O_* flags on Windows (and this can be enforced/engineered) and it provides an idiomatic way of specifying system-specific file opening behavior (insofar as os's POSIX-oriented interface can be considered idiomatic on Windows). This is the same route that's already used to pass in system-specific file opening behavior on POSIX platforms (e.g. syscall.O_DIRECTORY on Linux).

Even if the decision is made not to turn it on by default (which I think is the right call), I do agree that the flag has its use cases (including those pointed out by @networkimprov) and that it would be useful to be able to turn it on without having to resort to CreateFileW calls and copying boilerplate code.

@ddebroy

This comment has been minimized.

Copy link

commented Oct 3, 2019

Please consider alternative [1] above which allows developers the option to set FILE_SHARE_DELETE when necessary and prevent forked chunks of Go standard library code.

@networkimprov

This comment has been minimized.

Copy link
Author

commented Oct 3, 2019

Rust also sets this flag by default, and allows you to switch off any of the FILE_SHARE_* options.
https://doc.rust-lang.org/std/os/windows/fs/trait.OpenOptionsExt.html

@rsc could I ask which of the technical arguments made against the proposal were not refuted?

@DmitriyMV

This comment has been minimized.

Copy link

commented Oct 3, 2019

@networkimprov after reading the whole proposal and discussion I think you need to close this and re-open a new issue explicitly about adding new windows specific flag to os\syscall package. This would ensure that the new behavior is opt-in rather than opt-out. Current proposal implies changing existing code, which worries people.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2019

@rsc could I ask which of the technical arguments made against the proposal were not refuted?

I'm sorry, but that's not how the proposal process works. It is not enough to "refute" other people's arguments. "The goal of the proposal process is to reach general consensus about the outcome in a timely manner." There is no clear consensus about the path forward here. Technical arguments have been made, and they did not convince key Go developers who have made significant contributions to the Windows port (namely, @alexbrainman and @mattn). In addition to the lack of clear consensus, there is no clear sign of urgency to do something today: Go worked fine on Windows for nearly 10 years before this issue was filed. As I understand it, leaving everything alone means Go will continue to work as well as it always has.

I filed #34681 to provide an easier way to open files with FILE_SHARE_DELETE in the (still likely) event that this one is declined. It seemed more helpful to start a new thread limited to that idea than to continue this one, which has gotten very long.

@networkimprov

This comment has been minimized.

Copy link
Author

commented Oct 4, 2019

@rsc, before you decline this, let's hear what Alex thinks of your O_ALLOW_DELETE proposal.

Early in this discussion, he was against a new os.OpenFile() flag that sets file_share_delete, for the same reasons he disagreed with your os.Create/Open/OpenFile() suggestion. He is concerned that other programs assume that no one ever opens files that way, because MSVC fopen() cannot do so. Those (still unspecified) programs will therefore break Go programs which set the flag.

@alexbrainman

This comment has been minimized.

Copy link
Member

commented Oct 6, 2019

Alex, one can loop the rename attempt if outside access is allowed, ...

If you are prepared to loop the rename, you don't need to change anything in Go repo code. Your program will work as good as it works now.

Right now, the only solution is to literally copy a bunch of code out of Go's standard library, change one line, and then maintain that fork forever.

I think copying code into separate package is fine. While investigating moby/moby#39974 I got the same idea. I don't think there is much code there to maintain. In fact I implemented just that

https://github.com/alexbrainman/goissue34681

Feel free to copy or use as is. Maybe, given, there is so much interest in this functionality, you actually create proper package that can be shared and used by others. This way you can keep it up to date and fix bugs.

@rsc, before you decline this, let's hear what Alex thinks of your O_ALLOW_DELETE proposal.

Please, try

https://github.com/alexbrainman/goissue34681

first. If you not satisfied for some reason, we could discuss adding new functionality to Go repo.

Thank you.

Alex

@cpuguy83

This comment has been minimized.

Copy link

commented Oct 6, 2019

@alexbrainman

This comment has been minimized.

Copy link
Member

commented Oct 8, 2019

I'm really disappointed by this response.

I am sorry you feel that way. But did you actually tried to use my package?

Alex

@networkimprov

This comment has been minimized.

Copy link
Author

commented Oct 8, 2019

@rsc, since Alex is also opposed to an os.OpenFile() flag, should we do nothing?

How about putting this feature behind a build flag?

As to whether "Go worked fine on Windows for nearly 10 years," it definitely did not, in the event you needed to rename an open file. (It was also broken on Windows 8/10 laptops for the past 7 years.)

@cpuguy83

This comment has been minimized.

Copy link

commented Oct 8, 2019

@zx2c4

This comment has been minimized.

Copy link
Contributor

commented Oct 9, 2019

I'd suggest this specific proposal be declined. We're not going to introduce a bunch of TOCTOU security bugs for Windows users who relied on the existing behavior of os.OpenFile.

The larger question here is, "how can we expose to Go users the large variety of interesting flags for Windows' CreateFile and NtCreateFile functions?" I expect we'll be able to address those in different proposals, but certainly not by turning them all on by default as this here suggests.

@networkimprov

This comment has been minimized.

Copy link
Author

commented Oct 9, 2019

@zx2c4 did you read the whole thread? We weren't able to identify a single case of a Go user who relies on the existing, undocumented behavior, despite repeated attempts.

@andybons andybons modified the milestones: Go1.14, Proposal Oct 9, 2019
@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 9, 2019

It's been a week since #32088 (comment), and there is still very much no clear consensus, which means we should decline this.

Declined.

@rsc rsc closed this Oct 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.