Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prototype: Multiple language support with Rust #520

Closed
wants to merge 12 commits into from

Conversation

@andygrunwald
Copy link
Contributor

commented Jan 20, 2018

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • I have read the CONTRIBUTING document.
  • make ci passes on my machine.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.

Why is it useful?

This is a prototype version of #496.
This version is far from complete, but it should start a first discussion or feedback round.
The feedback round is important to let me / us know if it is useful to put more effort into it or to drop the idea at all.

This PR shows how this can be done for Rust.
Testing would be welcomed.

What it does?

It compiles a rust app with cargo build and adds it to the Artifacts.

How can you test it?

  1. Install Rust (e.g. via https://rustup.rs/)
  2. Kickstart a new project via cargo new --bin hello
  3. Place a new .goreleaser.yaml with this content into the project
rust:
- target:
  - x86_64-apple-darwin
  binary: hello

archive:
  format: tar.gz
  name_template: "{{ .ProjectName }}_{{ .Version }}_{{ .Os }}"
  1. Tag it and release it like every other tool

What is missing?

  • Unit tests
  • Proper testing with other pipes

Challenges

  • Goos, Goarch and Goarm are spreaded accross the codebase and used in a specific way (e.g. for templating). In the Rust context this adds a challenge, because targets are not splitted into the specific parts. They are represented like one string (e.g. x86_64-apple-darwin for Mac) known from LLVM
  • Copy / Paste code: Right now we just copied many of the code parts. Reusage would make sense here
andygrunwald added 11 commits Jan 20, 2018
feat: First prototype to compile Rust binaries
goreleaser is doing a lot of work that is not dependend
on the language itself. So lets use this functionality
and lets try to compile rust binaries with it.

See #496
feat: Support hooks for Rust pipe
In the new Rust pipe, this commit adds Hook support.
The implementation was copied from the Build pipe.
refactor: Extract code in Rust pipe to own functions
We organize the code a little bit more structured.
E.g. copyBinary into an own function
feat: Windows extension support for Rust pipe
If you enter a windows related target in the
Rust configuration, the extension ".exe" is
appended.

See #496
feat: Add defaults for Rust pipe
A default was added for rust.Binary.

See #496
fix: Add check if targets are configured in Rust pipe
The Rust pipe needs to have configured targets.
Otherwise the pipe is not functionable.

See #496
fix: errcheck analyzer for .Close() commands
The code analyzers don't like it when you .Close() a resource
but don't check the returning error.
In this case we ignore it.

See #496
fix: Set goarch empty in Rust pipe
Goos and Goarch are hardcoded into the tool.
For Rust we don't have the split here.
Just set Goos and empty Goarch.

See #496
docs: Write first version of docs for Rust pipe
A new pipe for Rust was introduced.
This commit adds the docs to use this.

See #496
style: Added comments for category pipes
When this tool goes multilingual,
comments for pipe categories make sense

See #496
fix: Check if there are more than 0 builds configured
Go builds are optional.
But there can be more builds than one.

See #496
@andygrunwald andygrunwald referenced this pull request Jan 20, 2018
2 of 2 tasks complete
@caarlos0

This comment has been minimized.

Copy link
Member

commented Jan 21, 2018

I like the idea but I'm not sure we should do it this way.

goreleaser already does too much (as someone else already stated in twitter).

I'm thinking that maybe we should have some sort pluggable pipes...

examples:

  • a piper that build for rust (or maybe in this case it could even be a builder instead of a piper
  • a piper that publishes new releases to twitter
  • a piper that generates the | sh shell scripts

I don't yet know how to make all this though.

I like the way terraform works, for example, and I think it's a good place to start studying the implementation...

@caarlos0

This comment has been minimized.

Copy link
Member

commented Jan 21, 2018

what do you think?

@caarlos0

This comment has been minimized.

Copy link
Member

commented Jan 21, 2018

thinking about a builder interface, we could have an interface like this:

type Builder interface {
	Build(ctx *context.Context, build config.Build, target buildtarget.Target) error
}

and then on the build pipe, something like:

var builder Builder
switch lang {
case "go":
	builder = gobuilder.Default
case "rust":
	builder = rustbuilder.Default
}

or maybe even better: builders register themselves:

package gobuilder

func init() {
	build.Register("go", Default)
}

and we n do something like this in the build pipe:

build.ForLang(lang).Build(/* ... */)

what do you think?

@caarlos0

This comment has been minimized.

Copy link
Member

commented Jan 21, 2018

I've prototyped something on #522

check it out @andygrunwald !

@caarlos0 caarlos0 referenced this pull request Jan 22, 2018
0 of 8 tasks complete
@caarlos0

This comment has been minimized.

Copy link
Member

commented Jan 22, 2018

about the goos, goarch and goarm on the artifact struct...

I think we can change them to uname-relatives. They are used to filter artifacts on other pipes (eg: docker, fpm, brew).

question is: can we break the rust target into os + arch + arm reliably?

@caarlos0

This comment has been minimized.

Copy link
Member

commented Jan 22, 2018

question is: can we break the rust target into os + arch + arm reliably?

looking into the results of rustc --print target-list, seems like we can!

// One idea would be, if no rust.Target are defined
// to add the default target for the current OS.
// E.g. hit `rustup show`. The default host is shown.
// For an up to date Mac is will show "Default host: x86_64-apple-darwin"

This comment has been minimized.

Copy link
@caarlos0

caarlos0 Jan 22, 2018

Member

I think we could default this to the same targets as go:

  • x86_64-apple-darwin
  • x86_64-unknown-linux-gnu
  • i686-apple-darwin
  • i686-unknown-linux-gnu

This comment has been minimized.

Copy link
@andygrunwald

andygrunwald Jan 22, 2018

Author Contributor

We thought about this as well. Like If MacOSx than target this. If linux, then this and so on. But we have to take the architecture and even the linker-system into account (when it comes to GNU libc or Musl libc). And there it is getting hard to decide what is the default

This comment has been minimized.

Copy link
@mre

mre Jan 22, 2018

Won't be easy. See comment here.

@andygrunwald

This comment has been minimized.

Copy link
Contributor Author

commented Jan 22, 2018

goreleaser already does too much (as someone else already stated in twitter).

I agree here. When we continue this road, it will become hard to maintain.

I like your idea of the "builder" and "piper" examples.
To generalize it even more: What about languages that are target independent? I think about JavaScript, Python (e.g. egg files), PHP (composer, phar), Java, Kotlin, Scala (JVM).
They have only one target.

I like the way terraform works, for example, and I think it's a good place to start studying the implementation...

Never looked deep into it. So at the moment, i am not a help here :(

about the goos, goarch and goarm on the artifact struct...
question is: can we break the rust target into os + arch + arm reliably?
looking into the results of rustc --print target-list, seems like we can!

Those targets are not targets created by rust. Those are targets from LLVM.
I didn't found a document what describes how to parse this, but there seems to be some kind of rule. A few examples:

aarch64-linux-android
aarch64-unknown-freebsd
aarch64-unknown-fuchsia
aarch64-unknown-linux-gnu
aarch64-unknown-linux-musl
asmjs-unknown-emscripten
i586-unknown-linux-gnu

Here are some slides that named it "target-triple" with Target Triple: <arch><sub>-<vendor>-<sys>-<abi>: https://llvm.org/devmtg/2014-04/PDFs/LightningTalks/2014-3-31_ClangTargetSupport_LighteningTalk.pdf

Maybe @mre or @timglabisch knows more?

@mre

This comment has been minimized.

Copy link

commented Jan 22, 2018

Yup, target triples. More on that here and here.

Triples are usually formatted as follows: {arch}-{vendor}-{sys}-{abi}.
For example, the triple arm-unknown-linux-gnueabihf refers to the systems that share these characteristics:

  • architecture: arm.
  • vendor: unknown. In this case, no vendor was specified and/or is not important.
  • system: linux.
  • ABI: gnueabihf. gnueabihf indicates that the system uses glibc as its C standard library (libc) implementation and has hardware accelerated floating point arithmetic (i.e. an FPU).

The attentive reader might have noticed, that triples are not necessarily triples.

Also - contrary to Go - Rust can not cross compile to all platforms on every platfrom. For example, it's quite tricky to compile to Mac from a Linux machine. Most people use external services like Travis or AppVeyor for that. There's also trust, which provides templates for these services to make cross compiling easier, but it still requires manual steps.

So, I guess if we want to support Rust, we should be a bit more conservative with the build targets and only use the host triple as a default. We can always add more targets later if possible.

@caarlos0

This comment has been minimized.

Copy link
Member

commented Jan 23, 2018

so I guess we can default to the current "triplet" only...

@andygrunwald

This comment has been minimized.

Copy link
Contributor Author

commented Jan 28, 2018

@mre @caarlos0 How do we continue here?
#522 is merged (🎉 ), this was the basement for this, right?

Should we (i, @mre, @timglabisch) finish this (with your help)?
Would you (@caarlos0) build the first implementation to show how it is done?

How did we align with the triplet defaults? Or with the native cargo cmd?
@mre You mentioned something related to "Build Rust inside a Docker container" for Linux, right? Can you elaborate on this or show an example?
And do you see this as a must-have? Or do you consider this as a user problem?

@mre

This comment has been minimized.

Copy link

commented Jan 28, 2018

@mre You mentioned something related to "Build Rust inside a Docker container" for Linux, right? Can you elaborate on this or show an example?

There's cross and cargo-docker, which both use Docker for cross-compiling Rust code.

And do you see this as a must-have? Or do you consider this as a user problem?

It's completely optional, I guess. At a future stage, we could piggy-back on the above tools for cross-compling, if they are installed. Alternatively, we could integrate a Docker pipeline to be completely independent. A sort of middle-ground would be having our own public (Rust) images that can be used for cross-compling.

@andygrunwald

This comment has been minimized.

Copy link
Contributor Author

commented Jan 28, 2018

Alternatively, we could integrate a Docker pipeline to be completely independent. A sort of middle-ground would be having our own public (Rust) images that can be used for cross-compling.

You mean a docker-rust? or a docker lang impl? this would be smart.

@mre

This comment has been minimized.

Copy link

commented Jan 28, 2018

You mean a docker-rust? or a docker lang impl? this would be smart.

Was thinking about a pure, language-agnostic docker pipeline. The user would define, which image will be used for compiling. Mostly, people might want to use the official images like rust or go for cross-compiling. Yes, even for Go it makes sense because not everybody might want to install the golang toolchain on their build server. Same for PHP + composer or Javascript + Yarn.
Just an idea. 😜

@caarlos0

This comment has been minimized.

Copy link
Member

commented Jan 28, 2018

#522 is merged (🎉 ), this was the basement for this, right?

yeap =D

Should we (i, @mre, @timglabisch) finish this (with your help)?
Would you (@caarlos0) build the first implementation to show how it is done?

I think you we can kind of do it together: change this to be a builder impl, we can change what else needs to be changed while doing it.

Meanwhile, I'll experiment with plugins and etc

@caarlos0

This comment has been minimized.

Copy link
Member

commented Jan 28, 2018

Was thinking about a pure, language-agnostic docker pipeline. The user would define, which image will be used for compiling. Mostly, people might want to use the official images like rust or go for cross-compiling. Yes, even for Go it makes sense because not everybody might want to install the golang toolchain on their build server. Same for PHP + composer or Javascript + Yarn.
Just an idea. 😜

like this idea as well =D

we could have a rust lang and a more generic docker lang (or docker-go, docker-rust etc)

@mre

This comment has been minimized.

Copy link

commented Jan 28, 2018

we could have a rust lang and a more generic docker lang (or docker-go, docker-rust etc)

Exactly. A pre-configured rust toolchain using cargo that covers most use-cases and a language-agnostic Docker toolchain for total control, which can be used for cross-compiling.

@caarlos0

This comment has been minimized.

Copy link
Member

commented Jan 28, 2018

or maybe we can do the plugin thing later (hopefully go will have native plugins for all platforms soon enough?)?

@andygrunwald

This comment has been minimized.

Copy link
Contributor Author

commented Mar 2, 2018

A small update here:
This prototype is not valid anymore. @caarlos0 implemented the base work in #522.
Now we or someone has to make the effort in implementing a new language/pipeline to support more than Go.

I think we should keep this PR open until we are able to manage it.
@caarlos0 @mre What do you think?

@guidiego

This comment has been minimized.

Copy link

commented Apr 1, 2018

Some news about use goreleaser + cargo?

@caarlos0

This comment has been minimized.

Copy link
Member

commented Apr 1, 2018

@caarlos0 @mre What do you think?

no problem on my side

@caarlos0

This comment has been minimized.

Copy link
Member

commented Apr 1, 2018

Some news about use goreleaser + cargo?

not at the moment :/

@andygrunwald

This comment has been minimized.

Copy link
Contributor Author

commented Apr 1, 2018

From my side as well no progress so far.
Time is quite limited at the moment, to less time to wrap my head around :(
Sorry.

@caarlos0

This comment has been minimized.

Copy link
Member

commented Apr 1, 2018

From my side as well no progress so far.
Time is quite limited at the moment, to less time to wrap my head around :(
Sorry.

no problem, take your time :)

@caarlos0

This comment has been minimized.

Copy link
Member

commented May 27, 2018

will close this for now.

I'll try to work on a plugin system in the next few days.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
You can’t perform that action at this time.