Fork rakyll/magicmime to add openbsd support and perhaps support cross compilation.

[anoduck]

When I attempted to install pistol, which I am totally stoked about, I received an error concerning build constraints. Never seen this one before, and I suspect it is an unintended error.

# Command Exec
$ env GO111MODULE=on go get -u github.com/doronbehar/pistol/cmd/pistol 
# Output recieved
package github.com/doronbehar/pistol/cmd/pistol
	imports github.com/doronbehar/pistol
	imports github.com/rakyll/magicmime: build constraints exclude all Go files in $HOME/go/bin/pkg/mod/github.com/rakyll/magicmime@v0.1.0

To me at least, it seems odd that all Go files are excluded. Any suggestions would be nice... this is a weird one.

[doronbehar]

Can't reproduce here on NixOS...

[anoduck]

Can't reproduce here on NixOS...

That is nice, but labeling it as can't reproduce would not be a true reflection on the state of the issue at hand. The encountered bug is well documented numerous times on github, and is most notably referenced in issues in the golang repository. https://github.com/golang/go/ssues/24068 Where it is stated the issue reported originated from a Nixos system. So, you might familiarize yourself with it, in the event it impacts future work.

[doronbehar]

FYI I do try to cross compile this package via Nix or via pure GNU-Make and I fail... See #72

[anoduck]

After looking at the code, the suggested solution to the referenced issue, and the source code for magicmime, it is most likely an issue with magicmime. Magicmime does not begin its main go script with package main to identify that it is the primary script of it's package, nor does it provide a main function for Go to locate and load. I am not a Go programmer, but these two issues appear to be problematic for Go to facilitate. Or, at least, that is what I can discern from the output.

[anoduck]

If instead of loading magicmime as a dependency, you just included it in your source code for the script as a part of your package. Would that make any difference? I.E. Hard code the library in.

[doronbehar]

Thanks for the investigation :) It might help me a lot with #72, because I do encounter the error:

build constraints exclude all Go files...

as well when I play with CGO_ENABLED when I test the static compilation. The natural thing to do is to try to fix this in a fork of magicmime and then when we'll be positive that it's the cause of your issue and/or the static compilation issue, we can send a PR. Unfortunately, magicmime seems a bit abandoned (no comments to PRs), so these fork might become permanent. I'm pretty busy these days so it might be nice if you'd test this yourself since I can't reproduce your issue. If you are not a Go programmer and you'd like to touch it a bit it might be a good experience for you.

[anoduck]

You are correct about the project looking abandoned, which leads me more towards believing magicmime being the cause to the issue I am experiencing.
I have no experience in Go, JS, or Java. Nor, would I have any idea where to start. If this were a python script, I would simply string all the other functions together under func main (...), but I doubt this would be an effective solution in this situation, since it is unclear what order the functions are called.
I have forked the repo, and merged it with another repo which appeared to have the most reasonable revisions. Unfortunately, these revisions did not resolve the issue I am experiencing. I will put it on the back burner for now, since my workload is nearly insurmountable now as it is.

[davewongillies]

When I attempted to install pistol, which I am totally stoked about, I received an error concerning build constraints. Never seen this one before, and I suspect it is an unintended error.

# Command Exec
$ env GO111MODULE=on go get -u github.com/doronbehar/pistol/cmd/pistol 
# Output recieved
package github.com/doronbehar/pistol/cmd/pistol
	imports github.com/doronbehar/pistol
	imports github.com/rakyll/magicmime: build constraints exclude all Go files in $HOME/go/bin/pkg/mod/github.com/rakyll/magicmime@v0.1.0

To me at least, it seems odd that all Go files are excluded. Any suggestions would be nice... this is a weird one.

Can you supply the output of go env for environment where you're trying to build this? You'll only get that error if you have CGO_ENABLED=0 set in your environment. github.com/rakyll/magicmime needs to have CGO_ENABLED=1 set for it to build, along with some other build dependencies

Running the following was successful for me with golang 1.17.2

CGO_ENABLED=1 go install github.com/doronbehar/pistol/cmd/pistol@latest

[doronbehar]

On my machine:

$ go env | grep CGO
GCCGO="gccgo"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

Now this is added to the README.

[doronbehar]

Thanks for the report and the investigation!

[anoduck]

I discovered this as well when doing my preliminary research, but since CGO_ENABLED=1 was already set, it was instantly dismissed as a viable resolution.

Here are the CGO relevant variables:

GCCGO="gccgo"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

And, my entire go env output is posted below.

GO111MODULE="on"
GOARCH="amd64"
GOBIN="/home/USER/go/bin/bin"
GOCACHE="/home/USER/.cache/go-build"
GOENV="/home/USER/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="openbsd"
GOINSECURE=""
GOMODCACHE="/home/USER/go/bin/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="openbsd"
GOPATH="/home/USER/go/bin"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/openbsd_amd64"
GOVCS=""
GOVERSION="go1.17.3"
GCCGO="gccgo"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="1" #<--- Here
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1703268905=/tmp/go-build -gno-record-gcc-switches"

Furthermore, executing CGO_ENABLED=1 go install github.com/doronbehar/pistol/cmd/pistol@latest resulted in the same error: import github.com/rakyll/magicmime: build constraints exclude all Go files.

[davewongillies]

Magicmime depends on libmagic. Do you have that installed on what looks to be an openbsd system?

[anoduck]

Yes, it is installed.

I believe the issue is caused by magicmime not possessing a main function, which would allow, or at least make it easier, for pistol to call the library.
Regardless, I have opened an issue in the magicmime repo that has been open now for 12 days, and I have not received a response. Which leads me to believe development and up keep has either stopped or is paused at the moment.

[davewongillies]

I believe the issue is caused by magicmime not possessing a main function, which would allow, or at least make it easier, for pistol to call the library.

Pistol compiles successfully on other platforms with magicmime, so the lack of a main() function is not going to be the problem. magicmime is used as a library in pistol. Libraries/modules in Go don't need main functions. This is going to be something specific to you attempting to compile this on OpenBSD.

[anoduck]

I believe the issue is caused by magicmime not possessing a main function, which would allow, or at least make it easier, for pistol to call the library.

Pistol compiles successfully on other platforms with magicmime, so the lack of a main() function is not going to be the problem. magicmime is used as a library in pistol. Libraries/modules in Go don't need main functions. This is going to be something specific to you attempting to compile this on OpenBSD.

Your response does not follow historically documented precedence with the generation of the returned error. In the referenced issue at this beginning of this thread, you will find that this error historically appeared on Nixos first, and one of the suggested resolutions was to ensure the main file in a library used package main instead of package whatever. At the end of the thread, a workaround for nixos was added to the package in question.

https://github.com/golang/go/ssues/24068

Since the error has been generating on other operating systems, and there is nothing that would distinguish running magicmime on OpenBSD from running it on linux. I fail to see how this could be the case.

Have you tried to install pistol on OpenBSD?

[4a6f656c]

github.com/rakyll/magicmime has build constraints that only allow it to build on darwin, freebsd and linux:

https://github.com/rakyll/magicmime/blob/master/magicmime.go#L15

If you're not building on one of those platforms, you'll get the "build constraints exclude all Go files" error since there are no Go files that can be compiled or built for that platform.

Someone could try adding openbsd to the build constraints to see if it builds/passes tests.

[anoduck]

@4a6f656c Dude, you are so awesome. It took some finagling, but that appears to have done the trick. Thanks for answering the call for help. Greatly appreciate it.

No lie, I had tried this earlier, but made serious mistakes in procedural application. I had to wipe rakyll/magicmime and an earlier version my anoduck/magicmime off my system to prevent referencing it, then I had to modify all references to it in my clone of pistol. Finally, I had to cut a release of magicmime in github, generate the checksums for the go.sum file, and install magicmime and pistol with make install.

To say the least, this has been a learning experience for me, and will benefit my career later on in the future. I am very thankful to @doronbehar and @4a6f656c for the patience, knowledge, understanding, and kindness.

Thanks a lot guys!

[doronbehar]

Thanks a lot for participating @4a6f656c, that's greatly appreciated. I'll keep this issue open because we should probably fork magicmime in order to at least add to it the openbsd build constraint. Perhaps more magicmime PRs can be merged in our fork.

[anoduck]

Thanks a lot for participating @4a6f656c, that's greatly appreciated. I'll keep this issue open because we should probably fork magicmime in order to at least add to it the openbsd build constraint. Perhaps more magicmime PRs can be merged in our fork.

I merged several pre-existing forks of magicmime which possessed recent updates and appeared to have a sustained history of maintenance into my fork, and added the OpenBSD build constraints as well. I also modified the pistol source code to pull from this fork, but I have not set up my pistol repository for creation of pull requests.
If you want me to save you some time, in repayment for your kindness, I can complete the setup of the pistol repo and generate a pull request for it. Also, you are more than welcome to any and all source code or references in my fork of magicmime.

Cheers.

[doronbehar]

@anoduck your work is appreciated. I forked magicmime here with a cleaner git commit history and a few more minor modifications.

I also finally made static compilation possible 🎉.

[anoduck]

Thanks for all your help and work. I will remove my repository for magicmime, clone your fork, and pull all the changes to pistol. Thanks again, and all the best.

[da-moon]

In case anyone is still getting issues with the linking, the problem might be due to custom linker flags in your cgo variables. Try using CGO_ENABLED=1 CGO_LDFLAGS="-g -O2" go install github.com/doronbehar/pistol/cmd/pistol@latest ; this fixed the build issue for me

[doronbehar]

Note also that now, thanks to Nix' wonderful cross compilation support, pistol's releases include assets of cross compiled packages for 3 common architectures. These may be of use to some of you.