Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use CI to build release binaries #200

Open
7 of 10 tasks
squeek502 opened this issue Mar 26, 2019 · 34 comments
Open
7 of 10 tasks

Use CI to build release binaries #200

squeek502 opened this issue Mar 26, 2019 · 34 comments

Comments

@squeek502
Copy link
Member

squeek502 commented Mar 26, 2019

Currently only have a few precompiled binaries for Luvi's newer releases. We've been meaning to set this up for a while but never gotten around to it. Example of when release deploying was added to luv via travis: luvit/luv@6b901a2

Related issues: #187, #181

Here's what we had in v2.7.6: https://github.com/luvit/luvi/releases/tag/v2.7.6

@creationix
Copy link
Member

Yes, this would be great if it could be automated. I think travis now has OSX guests and we have appveyor for windows, so it should be possible for most of these.

I was building freebsd and linux-686 in virtualbox images on my workstation, but I don't think those are as needed as others if it proves to be a problem.

@zhaozg
Copy link
Member

zhaozg commented Mar 27, 2019

Enough if win-x64,Mac-x64,Linux-x64 auto github-release for ci.Others are optional

@creationix
Copy link
Member

Sure, that would be the bare minimum. Next important on the list I think would be arm32 and arm64 variants. Those platforms are getting more and more popular.

@zhaozg
Copy link
Member

zhaozg commented Mar 27, 2019

That means arm platform need manual release, I do not have news about arm free CI platform

@creationix
Copy link
Member

Sometimes you can build them inside docker or chroots using qemu-user-static to emulate arm CPUs

@squeek502
Copy link
Member Author

I'm going to start working on the Windows targets now. Will do the Linux/Mac ones after that.

squeek502 added a commit to squeek502/luvi that referenced this issue Mar 27, 2019
Only does the deploy phase for tagged commits

Contributes towards luvit#200
rphillips pushed a commit that referenced this issue Mar 27, 2019
Only does the deploy phase for tagged commits

Contributes towards #200
@squeek502
Copy link
Member Author

squeek502 commented Mar 29, 2019

For Linux:

For Mac:

  • Right now we don't have Mac CI turned on at all, and I don't have a Mac to do any testing, so will probably need some help with this when I get around to it

For ARM:

For FreeBSD:

squeek502 added a commit to squeek502/luvi that referenced this issue Mar 29, 2019
Only does the deploy phase for tagged commits

Contributes towards luvit#200
@zhaozg
Copy link
Member

zhaozg commented Mar 29, 2019

@squeek502 I'll continue to finish Mac base your code.

@zhaozg
Copy link
Member

zhaozg commented Mar 29, 2019

The PR #203 add auto publish for osx.

here is ci log. here is real effect

Notice: travis need GITHUB_OAUTH_TOKEN to do deploy.

@squeek502
Copy link
Member Author

squeek502 commented Mar 29, 2019

Nice, thanks @zhaozg. Should we release v2.9.3 to try out all the new automatic publishing?

EDIT: Probably makes sense to merge #204 before a new release

@squeek502
Copy link
Member Author

squeek502 commented Mar 30, 2019

Here goes nothing:

EDIT: Appveyor failed due to auth_token:

Error creating GitHub release: Provider setting not found or it's value is empty. If secure setting is used please check that value was encrypted (or YAML was exported) while being logged under correct account.
Parameter name: auth_token

Either I messed something up or we'll need to use an auth token from a different account than mine.

@zhaozg
Copy link
Member

zhaozg commented Mar 30, 2019

I do git tag -d v2.9.3 && git tag v2.9.3 -m "v2.9.3" && git push origin v2.9.3 --force

@squeek502
Copy link
Member Author

I do git tag -d v2.9.3 && git tag v2.9.3 -m "v2.9.3" && git push origin v2.9.3 --force

I went with a different approach, and appveyor seems to be working now: https://github.com/luvit/luvi/releases/tag/v2.9.3-temp

Will clean things up once the build finishes: I'll move the Windows binaries to the v2.9.3 tag and delete the v2.9.3-temp tag. Then I'll revert c6a410e

@zhaozg
Copy link
Member

zhaozg commented Mar 30, 2019

Sorry for that, I will do nothing in next 4 hours. Please re tag v2.9.3 --force

@squeek502
Copy link
Member Author

Ok, I think we're good to go. The Windows binaries have been moved to the v2.9.3 tag and still have the proper version (luvi.exe v2.9.3), and the v2.9.3-temp tag has been deleted. The v2.9.3 tag seems fine to me, please verify that it's ok. And any future releases should work correctly (I think what happened is that I accidentally had copied some whitespace with the auth_token I encrypted the first time).

Here's an example of the random build error I mentioned in #202 btw: https://travis-ci.org/luvit/luvi/jobs/513318947

@zhaozg
Copy link
Member

zhaozg commented Mar 30, 2019

Checked package download from https://github.com/luvit/luvi/releases/tag/v2.9.3 test pass.

  • luvi-regular-Darwin_x86_64
  • luvi-regular-Linux_i686
  • luvi-regular-Linux_x86_64
  • luvi-regular-Windows-amd64.exe
  • luvi-regular-Windows-amd64.lib
  • luvi-regular-Windows-ia32.exe
  • luvi-regular-Windows-ia32.lib
  • luvi-tiny-Darwin_x86_64
  • luvi-tiny-Linux_i686
  • luvi-tiny-Linux_x86_64
  • luvi-tiny-Windows-amd64.exe
  • luvi-tiny-Windows-ia32.exe

@squeek502
Copy link
Member Author

Checked package download from https://github.com/luvit/luvi/releases/tag/v2.9.3 test pass.

Tested the rest except the Windows .lib files. Everything seems good.

@zhaozg
Copy link
Member

zhaozg commented Mar 30, 2019

@squeek502
Copy link
Member Author

Using docker & holy-build-box has portability benefits: https://github.com/phusion/holy-build-box

@creationix
Copy link
Member

I've had good experience using docker combined with qemu-user-static to build stuff in arm containers on x86 kernels/hosts

@squeek502
Copy link
Member Author

Looks like we can use Github actions to build arm binaries. @truemedian is doing just that here:

https://github.com/truemedian/luvit-bin/blob/main/.github/workflows/manual.yml#L154-L196

@truemedian
Copy link
Member

My approach in luvit-bin for ARM binaries is.. not the best, nor would I recommend it. It requires running a full system vm on github's actions machines, which turns a 2-3 minute build process into a 30-40 minute build process

@squeek502
Copy link
Member Author

squeek502 commented Mar 21, 2021

Surely it's better than nothing. If we don't want to take the hit for every CI, we could only build arm for tagged releases.

@truemedian
Copy link
Member

truemedian commented Mar 23, 2021

I decided to take another stab at this for building just luvi (this can be rather easily extended to both lit and luvit if need be).

ARM + OpenSSL binaries take the longest, so thats what these times are going to based off of.

from scratch: 37m 31s.
full ccache: 9m 56s.

Surprisingly, this is about on par with the current CI setup in terms of speed (when the ccache is full, having to compile slows down the process massively for ARM builds)

This allows for multiple things:

  • All (or most of) the CI in one place.
  • Running tests on PRs to prevent serious breakages.
  • +armv6/armv7 builds with "minimal effort"

A few things of note:

  • arm64v8 (aarch64) is currently unbuildable due to some issue in arm64 luajit (anyone want to pin it down? its a segfault / "attempt to compare two table values", possibly related)
  • no FreeBSD support

@SinisterRectus
Copy link
Member

I think we're ready to release 2.15.0, which will use Actions to build the release binaries.

Following this, I believe we can close #284, #275, and #279 and address #238, #266 and #280.

@Bilal2453
Copy link
Contributor

Honestly, one thing I wanted to change before releasing 2.15.0 is updating miniz to latest master branch, but I am having issues building and testing it so cba at the moment

@Bilal2453
Copy link
Contributor

Bilal2453 commented Jul 10, 2024

Should be ready now, miniz and LPeg have been updated to latest.

@SinisterRectus
Copy link
Member

SinisterRectus commented Jul 10, 2024

Indeed. As for the remaining checkboxes:

We now support aarch64 / arm64, but I guess we have to drop arm6 and arm7?

FreeBSD is supported but not with Actions.

@truemedian
Copy link
Member

We don't necessarily have to drop armv6 and armv7, however there are very few existing and maintained projects aimed towards wide glibc support. I didn't include them in my CI overhaul because I couldn't find a nice solution towards the maintenance of such an environment.

However, if one exists then it would be relatively trivial to add it to the build process.

@zhaozg
Copy link
Member

zhaozg commented Jul 11, 2024

I didn't include them in my CI overhaul because I couldn't find a nice solution towards the maintenance of such an environment.

Try download arm cross build chains from https://developer.arm.com/downloads/-/gnu-a.

@truemedian
Copy link
Member

There are two problems there:

We need to build with an old enough version of glibc that it will work on practically any distro using luvi.

We cannot cross compile. CMake does not understand the difference between host and target binaries. This distinction is required to build Luajit, and our build process is not suited to cross compile OpenSSL.

@zhaozg
Copy link
Member

zhaozg commented Jul 11, 2024

Yes, We need to build cmake toolchains config for that, I think consider it as long time tasks, not done that in 2.15.0

@truemedian
Copy link
Member

Just took a look around and https://github.com/vmactions/freebsd-vm seems it may be a decent fit for building FreeBSD binaries, the same organization also has similar actions for openbsd, solaris and dragonfly if we'd like to support those

@Bilal2453
Copy link
Contributor

Bilal2453 commented Aug 16, 2024

The rest of remaining architectures are of lower-priority to a lot of people, I believe. Supporting arm64 is the most important one beside the other major architectures because of the new ARM chips, I am not sure if Raspberry Pi is arm64 or armv7 but that could potentially influence the decision.

We can probably take a similar approach for armv6 and armv7, having QEMU do its magic.

I think it is fine to have a luvi release as of the current state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants