Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

syscall: UtimesNano does not use nanosecond system call on BSD, Solaris #16480

Closed
calmh opened this issue Jul 23, 2016 · 14 comments
Closed

syscall: UtimesNano does not use nanosecond system call on BSD, Solaris #16480

calmh opened this issue Jul 23, 2016 · 14 comments

Comments

@calmh
Copy link
Contributor

@calmh calmh commented Jul 23, 2016

ZFS on Solaris retains nanosecond precision file timestamps. These can be seen in the information returned by os.Stat. However os.Chtimes seems limited to setting microsecond precision. This means, among other things, that a file can't be copied with the timestamp fully retained.

jb@zlogin4:~/s/g/s/s/test $ go version
go version go1.7rc3 solaris/amd64
# Nanoseond precision before running program:
jb@zlogin4:~/s/g/s/s/test $ touch test.file 
jb@zlogin4:~/s/g/s/s/test $ stat test.file
  File: ‘test.file’
  Size: 0               Blocks: 1          IO Block: 131072 regular empty file
Device: 5a00010023h/386547122211d       Inode: 96951       Links: 1
Access: (0644/-rw-r--r--)  Uid: (  502/      jb)   Gid: (    1/   other)
Access: 2016-07-23 20:57:53.896596091 +0200
Modify: 2016-07-23 20:57:53.896596091 +0200
Change: 2016-07-23 20:57:53.896613387 +0200
 Birth: -

# Go program sees correct precision, attempts to set the same:
jb@zlogin4:~/s/g/s/s/test $ go run testchtimes.go test.file
2016-07-23 20:57:53.896596091 +0200 CEST

# Precision was reduced
jb@zlogin4:~/s/g/s/s/test $ stat test.file
  File: ‘test.file’
  Size: 0               Blocks: 1          IO Block: 131072 regular empty file
Device: 5a00010023h/386547122211d       Inode: 96951       Links: 1
Access: (0644/-rw-r--r--)  Uid: (  502/      jb)   Gid: (    1/   other)
Access: 2016-07-23 20:57:53.896596000 +0200
Modify: 2016-07-23 20:57:53.896596000 +0200
Change: 2016-07-23 20:58:02.133653117 +0200
 Birth: -

# The program:
jb@zlogin4:~/s/g/s/s/test $ cat testchtimes.go 
package main

import (
        "fmt"
        "os"
)

func main() {
        info, err := os.Stat(os.Args[1])
        if err != nil {
                fmt.Println(err)
                os.Exit(1)
        }
        t := info.ModTime()
        fmt.Println(t)
        if err := os.Chtimes(os.Args[1], t, t); err != nil {
                fmt.Println(err)
                os.Exit(1)
        }
}
@quentinmit quentinmit added this to the Go1.8 milestone Jul 23, 2016
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jul 24, 2016

I don't think there's anything Solaris-specific here.

The os package does the same thing on all POSIX systems (file_posix.go):

func Chtimes(name string, atime time.Time, mtime time.Time) error {
        var utimes [2]syscall.Timespec
        utimes[0] = syscall.NsecToTimespec(atime.UnixNano())
        utimes[1] = syscall.NsecToTimespec(mtime.UnixNano())
        if e := syscall.UtimesNano(name, utimes[0:]); e != nil {
                return &PathError{"chtimes", name, e}
        }
        return nil
}

... where syscall.UtimesNano always converts to microseconds. Many operating systems have utimens etc support but we don't appear to ever use it.

The os.Chtimes documentation is a little misleading:

The underlying filesystem may truncate or round the values to a less precise
time unit. If there is an error, it will be of type *PathError.

... since it's not really the filesystem truncating things.

This seems fixable.

@bradfitz bradfitz removed the OS-Solaris label Jul 24, 2016
@bradfitz bradfitz changed the title os: Chtimes doesn't retain full timestamp granularity on Solaris os: Chtimes doesn't use utimens, retain full timestamp granularity Jul 24, 2016
@odeke-em
Copy link
Member

@odeke-em odeke-em commented Sep 27, 2016

Currently only the syscall Linux-es' packages use utimensat in syscall.UtimesNano so that
nano-second precision is kept for the Linux-es but not other OSes. You can see that this is true on the Linuxes from the playground here https://play.golang.org/p/Jwg_7NW2gx and below from this search

Emmanuels-MBP-2:syscall emmanuelodeke$ go version
go version devel +5df7f52 Sun Sep 25 23:42:59 2016 +0000 darwin/amd64
Emmanuels-MBP-2:syscall emmanuelodeke$ grep -Rn 'func utimensat' *
zsyscall_linux_386.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_amd64.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_arm.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_arm64.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_mips64.go:126:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_mips64le.go:126:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_ppc64.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_ppc64le.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_s390x.go:126:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {

Unfortunately doesn't seem like darwin doesn't support utimensat and there is a comment in the syscall code about it
screen shot 2016-09-27 at 1 06 43 am

However, it seems like we can do this for Solaris and perhaps the other BSDs.
@mdempsky could you help me confirm whether this is still true and if utimensat can be supported on the other BSDs?

@quentinmit quentinmit added the NeedsFix label Oct 10, 2016
@rsc
Copy link
Contributor

@rsc rsc commented Oct 18, 2016

I think the package docs for os.Chtimes are correct. For example you can use a nanosecond-specific system call but if you're writing to FAT32, it's still going to get truncated to 2-second granularity.

The expectation is that the implementation of syscall.UtimesNano uses the most precise system call available. That's true on Linux: it tries utimensat and falls back to utimes when utimensat is not available. It appears to be true on Windows too: it uses SetFileTime which works at 100ns granularity, but that's true of everything on Windows.

On BSD systems, there is a TODO in syscall_bsd.go to use a higher precision call.

On Solaris, there is no TODO but still the implementation uses plain syscall.Utimes. If Solaris has a more precise call, the Solaris implementation of syscall.UtimesNano should use it.

@rsc rsc changed the title os: Chtimes doesn't use utimens, retain full timestamp granularity syscall: UtimesNano does not use nanosecond system call on BSD, Solaris Oct 18, 2016
@mdempsky
Copy link
Member

@mdempsky mdempsky commented Oct 18, 2016

@odeke-em utimensat has been supported since OpenBSD 5.0 (Nov 2011), NetBSD 6.0 (Oct 2012), and FreeBSD 11.0 (Oct 2016). It looks like it's still unsupported on Darwin though.

@odeke-em
Copy link
Member

@odeke-em odeke-em commented Oct 29, 2016

Thanks for the clarification @mdempsky!

@bradfitz I don't know who to ask but might you access to some Solaris and BSD machines?

@odeke-em
Copy link
Member

@odeke-em odeke-em commented Oct 29, 2016

Actually ama let someone who is an expert on syscalls and BSD handle this,
instead of me holding up the freeze cut-off that is in less than 30 hours.

@binarycrusader
Copy link
Contributor

@binarycrusader binarycrusader commented Oct 29, 2016

All actively supported versions of Solaris from Solaris 10 onwards support utimensat since roughly 2010.

@binarycrusader
Copy link
Contributor

@binarycrusader binarycrusader commented Oct 29, 2016

You can see that I added support (changed UtimesNano) to use this in the sys/unix package as part of the changes for #8609:

golang/sys@68a71b6

https://golang.org/cl/14643

@rsc rsc modified the milestones: Go1.9, Go1.8 Nov 11, 2016
@odeke-em
Copy link
Member

@odeke-em odeke-em commented Mar 21, 2017

Hello there @binarycrusader, the Go1.9 tree is open. Any interest/bandwidth for getting this in? Thanks.

@bradfitz bradfitz modified the milestones: Go1.10, Go1.9 Jun 7, 2017
@tklauser
Copy link
Member

@tklauser tklauser commented Aug 11, 2017

The same problem also exists in x/sys/unix. I'll have a look at it and send CLs for both.

tklauser added a commit to tklauser/go that referenced this issue Aug 11, 2017
All the BSDs and Solaris support the utimensat syscall, but Darwin
doesn't. Account for that by adding the //sys lines not to
syscall_bsd.go but the individual OS's syscall_*.go files and implement
utimensat on Darwin as just returning ENOSYS, such that UtimesNano will
fall back to use utimes as it currently does unconditionally.

This also adds the previously missing utimensat syscall number for
FreeBSD and Dragonfly.

Fixes golang#16480

Change-Id: I367454c6168eb1f7150b988fa16cf02abff42f34
@tklauser
Copy link
Member

@tklauser tklauser commented Aug 11, 2017

Submitted https://golang.org/cl/55130 attempting to fix this issue in the syscall package. So far, I was only able to do some rudimentary testing on FreeBSD so far and I'd appreciate if anyone running Solaris with a nanosecond-precision file system could give it a try (/cc @calmh @binarycrusader). The change is also available in tklauser/go@a17c114.

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 11, 2017

Change https://golang.org/cl/55130 mentions this issue: syscall: add utimensat and use it for UtimesNano on BSD and Solaris

@gopherbot gopherbot closed this in 67c3605 Aug 14, 2017
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 14, 2017

Change https://golang.org/cl/55141 mentions this issue: syscall: really use utimensat for UtimesNano on Solaris

gopherbot pushed a commit that referenced this issue Aug 15, 2017
golang.org/cl/55130 added utimensat for Solaris but didn't use it in
UtimesNano (despite indicating otherwise in the commit message). Fix
this by also using utimensat for UtimesNano on Solaris.

Because all versions of Solaris suppported by Go support utimensat,
there is no need for the fallback logic and utimensat can be called
unconditionally.

This issue was pointed out by Shawn Walker-Salas.

Updates #16480

Change-Id: I114338113a6da3cfcb8bca950674bdc8f5a7a9e5
Reviewed-on: https://go-review.googlesource.com/55141
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@golang golang locked and limited conversation to collaborators Aug 14, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.