How to add Xstdin, Xstdout and Xstderr?

[kortschak]

After merging #6, #7 and #8 I am able to go build the majority of the rtmp.go artifact produced by ccgo's work on librtmp.a. However, there is no symbol for the standard streams and so I see the following failure:

$ go build -gcflags=-e rtmp.go
# command-line-arguments
./rtmp.go:7599:37: undefined: crt.Xstderr
./rtmp.go:7814:37: undefined: crt.Xstderr
./rtmp.go:7864:37: undefined: crt.Xstderr

Looking at how the streams are handled, it's not obvious to me how crt.Xstd??? should be exposed; either the vars (stdin|stdout|stderr) could be renamed to X\1, or the constants X\1 could be created. Neither of these approach seem sensible or safe.

[cznic]

I'm aware of this problem and it's a bit complicated to solve. For historical reasons, support for C variables stdin, stdout, stderr was a hack working for standalone programs (commands) only. I've not anticipated they may be used in a package/library as well.

This can be fixed, but it will clash with my effort to port musl libc and use it instead of all the Go-simulates-libc hackery. I've spent some time to get a proof-of-concept working in the musl branch and I now believe it's feasible.

TL;DR: For now I suggest a bypass, kind of.

jnml@r550:~/tmp> cat main.c 
int main() {}
jnml@r550:~/tmp> ccgo -o main.go main.c
jnml@r550:~/tmp> cat main.go
// Code generated by 'ccgo -o main.go main.c', DO NOT EDIT.

// +build linux,amd64

package main

import (
	"math"
	"os"
	"unsafe"

	"github.com/cznic/crt"
)

const (
	null = uintptr(0)
)

var (
	_ = math.Pi
	_ = os.DevNull
	_ = unsafe.Pointer(null)

	nz32 float32
	nz64 float64
)

func init() { nz32 = -nz32 }
func init() { nz64 = -nz64 }

func alloca(p *[]uintptr, n int) uintptr   { r := crt.MustMalloc(n); *p = append(*p, r); return r }
func preinc(p *uintptr, n uintptr) uintptr { *p += n; return *p }

func main() {
	psz := unsafe.Sizeof(uintptr(0))
	argv := crt.MustCalloc((len(os.Args) + 1) * int(psz))
	p := argv
	for _, v := range os.Args {
		*(*uintptr)(unsafe.Pointer(p)) = crt.CString(v)
		p += psz
	}
	a := os.Environ()
	env := crt.MustCalloc((len(a) + 1) * int(psz))
	p = env
	for _, v := range a {
		*(*uintptr)(unsafe.Pointer(p)) = crt.CString(v)
		p += psz
	}
	*(*uintptr)(unsafe.Pointer(Xenviron)) = env
	X_start(crt.NewTLS(), int32(len(os.Args)), argv)
}

// linking main.o

// X__ccgo_va_end *void, escapes: true, builtin.h:36:6
var X__ccgo_va_end = bss + 0

// X__ccgo_va_start *void, escapes: true, builtin.h:37:6
var X__ccgo_va_start = bss + 8

// Xmain is defined at main.c:1:5
func Xmain(tls crt.TLS, _ int32, _ uintptr /* **int8 */) (r int32) {
	return r
}

// linking crt0.o

// X__stdfiles [3]uintptr, escapes: true, crt0.c:3:15
var X__stdfiles = bss + 16

// Xenviron **int8, escapes: true, crt0.c:4:6
var Xenviron = bss + 40

// Xstdin *void, escapes: true, crt0.c:5:6
var Xstdin = bss + 48 // pointer to void

func init() { *(*uintptr)(unsafe.Pointer(Xstdin)) = X__stdfiles }

// Xstdout *void, escapes: true, crt0.c:5:31
var Xstdout = bss + 56 // pointer to void

func init() { *(*uintptr)(unsafe.Pointer(Xstdout)) = X__stdfiles + 8 }

// Xstderr *void, escapes: true, crt0.c:5:57
var Xstderr = bss + 64 // pointer to void

func init() { *(*uintptr)(unsafe.Pointer(Xstderr)) = X__stdfiles + 16 }

// X_start is defined at crt0.c:7:6
func X_start(tls crt.TLS, _argc int32, _argv uintptr /* **int8 */) {
	crt.X__register_stdfiles(tls, *(*uintptr)(unsafe.Pointer(Xstdin)), *(*uintptr)(unsafe.Pointer(Xstdout)), *(*uintptr)(unsafe.Pointer(Xstderr)), Xenviron)
	crt.X__builtin_exit(tls, Xmain(tls, _argc, _argv))
}

func init() { *(*uintptr)(unsafe.Pointer(Xstdin)) = X__stdfiles }

func init() { *(*uintptr)(unsafe.Pointer(Xstdout)) = X__stdfiles + 8 }

func init() { *(*uintptr)(unsafe.Pointer(Xstderr)) = X__stdfiles + 16 }

var (
	bss     = crt.BSS(&bssInit[0])
	bssInit [72]byte
)
jnml@r550:~/tmp> 

I think adding a Go file to your project with content like this (not tested) should solve the problem:

// Bypass missing stdin, stdout and stderr

package rtmp

import (
	"os"
	"unsafe"

	"github.com/cznic/crt"
)

func init() {
	psz := unsafe.Sizeof(uintptr(0))
	argv := crt.MustCalloc((len(os.Args) + 1) * int(psz))
	p := argv
	for _, v := range os.Args {
		*(*uintptr)(unsafe.Pointer(p)) = crt.CString(v)
		p += psz
	}
	a := os.Environ()
	env := crt.MustCalloc((len(a) + 1) * int(psz))
	p = env
	for _, v := range a {
		*(*uintptr)(unsafe.Pointer(p)) = crt.CString(v)
		p += psz
	}
	*(*uintptr)(unsafe.Pointer(Xenviron)) = env
	X_start(crt.NewTLS(), int32(len(os.Args)), argv)
}

// linking crt0.o

// X__stdfiles [3]uintptr, escapes: true, crt0.c:3:15
var X__stdfiles = bss2 + 16

// Xenviron **int8, escapes: true, crt0.c:4:6
var Xenviron = bss2 + 40

// Xstdin *void, escapes: true, crt0.c:5:6
var Xstdin = bss2 + 48 // pointer to void

func init() { *(*uintptr)(unsafe.Pointer(Xstdin)) = X__stdfiles }

// Xstdout *void, escapes: true, crt0.c:5:31
var Xstdout = bss2 + 56 // pointer to void

func init() { *(*uintptr)(unsafe.Pointer(Xstdout)) = X__stdfiles + 8 }

// Xstderr *void, escapes: true, crt0.c:5:57
var Xstderr = bss2 + 64 // pointer to void

func init() { *(*uintptr)(unsafe.Pointer(Xstderr)) = X__stdfiles + 16 }

// X_start is defined at crt0.c:7:6
func X_start(tls crt.TLS, _argc int32, _argv uintptr /* **int8 */) {
	crt.X__register_stdfiles(tls, *(*uintptr)(unsafe.Pointer(Xstdin)), *(*uintptr)(unsafe.Pointer(Xstdout)), *(*uintptr)(unsafe.Pointer(Xstderr)), Xenviron)
}

func init() { *(*uintptr)(unsafe.Pointer(Xstdin)) = X__stdfiles }

func init() { *(*uintptr)(unsafe.Pointer(Xstdout)) = X__stdfiles + 8 }

func init() { *(*uintptr)(unsafe.Pointer(Xstderr)) = X__stdfiles + 16 }

var (
	bss2     = crt.BSS(&bssInit2[0])
	bssInit2 [72]byte
)

Please try and let me know the outcome, thank you.

[kortschak]

That looks like it is doing a few things repeatedly; assignment to Xstd??? happen twice in init funcs and also in var decls.

I will presumably need to edit the rtmp.go file to not try to get these from crt.

I'll let you know how it goes.

[cznic]

That looks like it is doing a few things repeatedly; assignment to Xstd??? happen twice in init funcs and also in var decls.

I don't think so. For example

var Xstdin = bss2 + 48
func init() { *(*uintptr)(unsafe.Pointer(Xstdin)) = X__stdfiles }

The variable initializer makes Xstdin point to bss2+48. The init function makes the pointer Xstdin points to point to where X__stdfiles points to. Sounds messy? Welcome to the world of C-to-Go translation ;-) Those additional indirection levels are caused by the fact that those variables do escape, ie. their address is taken.

I will presumably need to edit the rtmp.go file to not try to get these from crt.

I suggest to put the bypass in some, say, stdio.go file, assuming rtmp.go is the file produced by ccgo. They will be both part of the same package and not interfere with each other above one providing the missing definitions to the other. Just a normal multi file Go package where one of the files is produced mechanically by ccgo.

[kortschak]

That makes sense, but there are two lots of each of those init funcs, which doesn't make sense to me.

Yes, I tried having a file like that (bypass.go in my case) in the package with rtmp.go, but rtmp.go has references to crt.Xstd??? not Xstd???. This is straightforward to fix with sed.

I'll be taking the current state to our project meeting tomorrow so we can discuss the approaches we are going to take with this. I have to say I am sure that it will be helpful.

[cznic]

That makes sense, but there are two lots of each of those init funcs, which doesn't make sense to me.

Yep, I haven't noticed that and talked about a different thing, sorry. Indeed, that's linker failure (now cznic/ccgo#4).

This is straightforward to fix with sed.

I think that's a good way.

I'll be taking the current state to our project meeting tomorrow so we can discuss the approaches we are going to take with this. I have to say I am sure that it will be helpful.

I think we should remind ourselves that the crt/cc/ccgo combo is not well tested and battle-proved. I'd not recommend its use in a mission-critical project yet.

[kortschak]

I think we should remind ourselves that the crt/cc/ccgo combo is not well tested and battle-proved. I'd not recommend its use in a mission-critical project yet.

That's understood. Having spent some time looking through the generated code I suspect that it will work reasonably well as an intermediate to porting the code piecemeal.