Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: closing Darwin FIFO writer does not trigger kqueue event for reader #24164

Closed
stjohnjohnson opened this issue Feb 27, 2018 · 5 comments

Comments

Projects
None yet
4 participants
@stjohnjohnson
Copy link

commented Feb 27, 2018

What version of Go are you using (go version)?

go version go1.10 linux/amd64
go version go1.10 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What did you do?

Ran this script here that reads/writes from a FIFO. On Linux the reader receives the EOF. On Darwin it does not.

package main

import (
    "bufio"
    "fmt"
    "os"
    "time"
)

// @note This is used to get around bufio.Scanner erroring at large lines
func readln(r *bufio.Reader) (string, error) {
    var (
        isPrefix = true
        err      error
        line, ln []byte
    )

    for isPrefix && err == nil {
        line, isPrefix, err = r.ReadLine()
        ln = append(ln, line...)
    }

    return string(ln), err
}

func reader(filePath string) {
    f, err := os.Open(filePath)
    if err != nil {
        panic(err)
    }

    reader := bufio.NewReader(f)
    fmt.Print("READER >> created\n")

    line, readErr := readln(reader)

    for readErr == nil {
        fmt.Printf("READER >> read 1 line: %+v\n", line)
        line, readErr = readln(reader)
    }

    fmt.Printf("READER >> read finished: %+v\n", readErr)
}

func writer(filePath string) {
    f, err := os.OpenFile(filePath, os.O_WRONLY, 0600)

    fmt.Printf("WRITER << opened: %+v|%+v\n", f, err)
    if err != nil {
        panic(err)
    }

    fmt.Printf("WRITER << encoder created\n")

    for i := 0; i < 3; i++ {
        time.Sleep(1 * time.Second)
        _, err = f.WriteString(fmt.Sprint("line", i, "\n"))
        fmt.Printf("WRITER << written line%d, %+v\n", i, err)
    }

    time.Sleep(1 * time.Second)
    err = f.Close()
    fmt.Printf("WRITER << closed: %+v\n", err)
}

func main() {
    fifo := os.Args[1]

    fmt.Printf("STARTED %s\n", fifo)

    go writer(fifo)
    reader(fifo)

    fmt.Print("ALL DONE\n")
}

Linux

go version go1.10 linux/amd64
STARTED /tmp/tmp.6if4ZSWnZ8/fifo
READER >> created
WRITER << opened: &{file:0xc42008c000}|<nil>
WRITER << encoder created
WRITER << written line0, <nil>
READER >> read 1 line: line0
WRITER << written line1, <nil>
READER >> read 1 line: line1
WRITER << written line2, <nil>
READER >> read 1 line: line2
WRITER << closed: <nil>
READER >> read finished: EOF
ALL DONE

Darwin

go version go1.10 darwin/amd64
STARTED /var/folders/k_/zdt3jb2j7gx1rnf379k_51gh0000gn/T/tmp.ksHGLSvs/fifo
READER >> created
WRITER << opened: &{file:0xc42009e000}|<nil>
WRITER << encoder created
WRITER << written line0, <nil>
READER >> read 1 line: line0
WRITER << written line1, <nil>
READER >> read 1 line: line1
WRITER << written line2, <nil>
READER >> read 1 line: line2
WRITER << closed: <nil>
^C
signal: interrupt

Repo for reproduction: https://github.com/stjohnjohnson/golang-eof-fifo

Notes

Was able to reproduce it as far back as go1.9 darwin/amd64 using asdf. Last working version was go1.8.7 darwin/amd64.

@ianlancetaylor ianlancetaylor changed the title bufio.Reader.ReadLine not receiving EOF on OSX bufio: Reader.ReadLine from FIFO not receiving EOF on OSX Feb 28, 2018

@ianlancetaylor ianlancetaylor added this to the Go1.11 milestone Feb 28, 2018

@as

This comment has been minimized.

Copy link
Contributor

commented Feb 28, 2018

This is caused by the poller. Related issue #22024 that happened on Windows. Confirmed with git bisect

c05b06a12d005f50e4776095a60d6bd9c2c91fac is the first bad commit
commit c05b06a12d005f50e4776095a60d6bd9c2c91fac
Author: Ian Lance Taylor <iant@golang.org>
Date:   Fri Feb 10 15:17:38 2017 -0800

    os: use poller for file I/O
@as

This comment has been minimized.

Copy link
Contributor

commented Mar 3, 2018

Adding darwin to the conditional statement here makes your example work, but I don't know if that's a good idea.

@stjohnjohnson
You might be able to use this workaround https://play.golang.org/p/G8YIi1E7j4x. It calls Fd() on the reader's side of the pipe and enables blocking IO again... made the example work on my darwin system without modifying the go source.

@stjohnjohnson

This comment has been minimized.

Copy link
Author

commented Mar 6, 2018

@as that works! Thanks!

APShirley added a commit to APShirley/reconfigure-pipeline that referenced this issue Mar 15, 2018

Temporary fix for reconfigure-pipeline issue with go
golang/go#24164

Signed-off-by: Adrian Zankich <azankich@pivotal.io>

@ianlancetaylor ianlancetaylor changed the title bufio: Reader.ReadLine from FIFO not receiving EOF on OSX runtime: closing Darwin FIFO writer does not trigger kqueue event for reader Jun 13, 2018

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2018

I can confirm that on MacOS 16.3.0 closing the last writer to a FIFO does not trigger a kqueue event for the reader. In Go's implementation, that causes the reader to block indefinitely.

There is a Stack Overflow questions that mentions this, with no solution offered: https://stackoverflow.com/questions/49481723/difference-in-kqueue-handling-of-fifos-between-mac-os-and-freebsd .

My best suggestion at this point is that on Darwin we will have to avoid treating fifos as pollable.

@gopherbot

This comment has been minimized.

Copy link

commented Jun 13, 2018

Change https://golang.org/cl/118566 mentions this issue: os: don't poll fifos on Darwin

@gopherbot gopherbot closed this in 4a778cd Jun 13, 2018

dna2github added a commit to dna2fork/go that referenced this issue Jun 14, 2018

os: don't poll fifos on Darwin
The Darwin kqueue implementation doesn't report any event when the
last writer for a fifo is closed.

Fixes golang#24164

Change-Id: Ic2c47018ef1284bf2e26379f8dd7646edaad4d05
Reviewed-on: https://go-review.googlesource.com/118566
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.