Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/env/linux-mips: we have no Linux MIPS builders #31217

Closed
bradfitz opened this issue Apr 2, 2019 · 79 comments
Closed

x/build/env/linux-mips: we have no Linux MIPS builders #31217

bradfitz opened this issue Apr 2, 2019 · 79 comments

Comments

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Apr 2, 2019

All four of our MIPS variants have lost their builders and the email address of the old owner now bounces. (changed jobs, presumably)

So, we need to find a new owner, or remove the port per policy (https://github.com/golang/go/wiki/PortingPolicy#removing-a-port)

/cc @randall77 @cherrymui @ianlancetaylor @dmitshur

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 2, 2019

Change https://golang.org/cl/170444 mentions this issue: dashboard: update stale ownership info for now-dead MIPS builders

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Apr 2, 2019

I have a MIPS64LE machine that I used to run as a builder (old style builder, not buildlet). The machine is quite old and slow, so I stopped to run it once the new builders were up. I could probably revive that machine, if desired. (but probably no earlier than next month)

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Apr 2, 2019

Alternatively, I could run it for you if you want to ship or inter-office mail it to me.

Do you remember which board it is?

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Apr 2, 2019

It's a Loongson 2E box.

If I remember correctly it needs some awkward booting process, something like PXE with TFTP from another machine, because I screwed up something (probably part of the hard drive?) long time ago. Once it boots, it should run ok.

I also have a Loongson 2F laptop, which is probably a little better. I think it can boot by itself. But the disk space is pretty small.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Apr 2, 2019

Loading

@willglynn
Copy link

@willglynn willglynn commented Apr 2, 2019

@bradfitz Same board is available at a lower price and higher quantity elsewhere on eBay or from Mouser, a reputable electronics distributor.

Loading

@minux
Copy link
Member

@minux minux commented Apr 2, 2019

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Apr 2, 2019

@willglynn, thanks for the links! I've now bought all the ones I found on eBay. We'll see how those work out before I buy any more from Mouser.

Loading

gopherbot pushed a commit to golang/build that referenced this issue Apr 3, 2019
Updates golang/go#31217

Change-Id: I9b8e64c24ed303166e609c13fa4b69d83ddcd0c1
Reviewed-on: https://go-review.googlesource.com/c/build/+/170444
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@omac777
Copy link

@omac777 omac777 commented Apr 3, 2019

Alternatively, I could run it for you if you want to ship or inter-office mail it to me.

Do you remember which board it is?
I have a six-core Loongson 3a board 8GB RAM that used to have an old fedora running on it, but I blew it away while trying to bring debian unstable up onto it using mediatemple mips binaries onto it. I didn't get much support from loongson themselves because I'm not in China. I would need a new hard-drive with the bootable loongnix installed onto it to bring it back up and running enough to run golang on it. the debian loongson experts within qemu never got back to me. It's not a typical board where you can just boot with a cd or a usb drive. I'm not willing to part with this $$$ board, but if you could help me bring the box up again, I'll give you root access via fiber to it. That would be fair.

Loading

@cetinsert
Copy link

@cetinsert cetinsert commented Apr 3, 2019

Linux / MIPS is business critical for us! I can get all necessary hardware and setup a new repository for automated builds with unreleased versions of go.
Currently we use it for a fleet of IIoT devices with GOOS=linux GOARCH=mips with always the latest release of go (1.12 as of today). Hardware is all openwrt routers / router boards.

Loading

@cetinsert
Copy link

@cetinsert cetinsert commented Apr 3, 2019

@bradfitz I have a doctor's appointment for the next few hours but am ready to take a role as builder for all 4 MIPS variants. Please let me know what next to do; otherwise I will review posts and documents and get in touch with a hardware / time plan. I expect getting all necessary and future hardware (for the other 3 variants (we only use GOARCH=mips in automated builds)) will take mere days because I live in Korea, relatively close to Shenzhen.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Apr 3, 2019

@cetinsert, great to hear! How much RAM and what type of storage do those devices have? We tend to require 1GB of RAM and pretty decent storage (SD cards fail very quickly, so putting its working directory on a USB SSD disk is better). They also need to have moderately okay network, but if you're in South Korea you're probably good. :-)

https://golang.org/wiki/DashboardBuilders has more info. I can get you the necessary keys over email.

Loading

@cetinsert
Copy link

@cetinsert cetinsert commented Apr 3, 2019

@bradfitz ok so we were not talking about cross-compiling for mips o__O (because that's what we do from linux-amd64, darwin-amd64 and windows-amd64)?

Let me review https://golang.org/wiki/DashboardBuilders!

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Apr 3, 2019

@cetinsert, we kinda have support for that, at least partially. The builders support cross-compiling the make.bash step but when running tests it still does on-device compiles of each test, which can still be big. We don't yet have support for cross-compiling the test binaries. Even if we did, that'd increase the network requirements a bit.

Loading

@cetinsert
Copy link

@cetinsert cetinsert commented Apr 3, 2019

@bradfitz I see VMs listend here https://farmer.golang.org/builders#host-linux-mips. If VMs are ok, please send me the keys over email (mips@shin.asia)!

I have several Scaleway x86-64 (2 x amd64 cores, 2GB RAM) machines with good network conditions and can setup QEMU? and other essentials for MIPS emulation or take over existing work in the form of VM images and host them from then on.

Loading

@9nut
Copy link

@9nut 9nut commented Apr 3, 2019

@bradfitz I can contribute a Ci20; I'll order one today. If it's time critical, I have a mikrotik rb450g (256mb) and a vocore2 (128mb) that I can send your way; probably only useful for testing. Incidentally, the rb450g runs Plan 9 via tftp/bootp -- if/when plan9/mips32 is available.

Loading

@FlyGoat
Copy link

@FlyGoat FlyGoat commented Apr 3, 2019

@omac777

Alternatively, I could run it for you if you want to ship or inter-office mail it to me.
Do you remember which board it is?
I have a six-core Loongson 3a board 8GB RAM that used to have an old fedora running on it, but I

I assume you have a Loongson-3B1500 since Loongson-3A only have four cores. 3B1500 is a octa-core model but two cores are disabled on some boards due to a hardware errata.

blew it away while trying to bring debian unstable up onto it using mediatemple mips binaries onto it. I
didn't get much support from loongson themselves because I'm not in China. I would need a new
hard-drive with the bootable loongnix installed onto it to bring it back up and running enough to run
golang on it. the debian loongson experts within qemu never got back to me. It's not a typical board
where you can just boot with a cd or a usb drive. I'm not willing to part with this $$$ board, but if you
could help me bring the box up again, I'll give you root access via fiber to it. That would be fair.

Could you please tell me the model of your board? It should be displayed in firmware, or printed on circuit board. Probably you just need update PMON or klBIOS to make it boot from USB stick.

And now, we have a Fedora28 Port: http://mirror.lemote.com:8000/fedora/fedora28-live/
You can try to write it to a USB stick and boot or even write it directly to your hard drive.

Anyway, though I'm not a Loongson employee, I'm familiar with Loongson devices and Loongson developers, email me if you need any help.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Apr 3, 2019

@9nut, thanks, Skip, but I have a few Ci20s on the way already. No need to buy another. The rb450g could be interesting, though, as that's a 32-bit BE CPU it seems? Not much RAM, but it'd give us a MIPS CPU variant I don't think we have.

I don't think there are any plans for more plan9 ports (@0intro?). But really the first priority is getting Linux back.

Loading

@cetinsert
Copy link

@cetinsert cetinsert commented Apr 3, 2019

@bradfitz would QEMU or another mips emulator work?
If yes, I can dedicate an x86-64 host with 2GB RAM and SSD storage.

For a true mips host, we will be reviewing hardware options shortly.
Our mips-based target devices have nowhere near 1GB of RAM.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Apr 3, 2019

@cetinsert, QEMU is generally too slow and sometimes I hear it's too forgiving (e.g. accepting unaligned instructions that real hardware would reject). So we try to use real hardware when possible. If we do decide to go the emulation route we can run that on GCE where we have tons of x86 capacity.

Loading

@0intro
Copy link
Member

@0intro 0intro commented Apr 3, 2019

Currently, Plan 9 runs on MikroTik RB450G (MIPS32 BE) and
Lemote Yeeloong (MIPS64 LE). OCTEON (MIPS64 BE) and CI20
(MIPS32 LE) ports were in progress.

However, there is currently no plan to port Go on plan9/mips,
thought it could be interesting.

Anyway, I think these board would likely be a little tight
to run a (Linux) Go builder.

Ideally, you could try to get your hands on an OCTEON board,
which usually have multiple cores and multiple gigabytes of memory.

Loading

@cetinsert
Copy link

@cetinsert cetinsert commented Apr 3, 2019

@bradfitz which hardware was used by the former builders for the 4 variants for Linux?

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Apr 3, 2019

@cetinsert, I don't actually know. It was run by somebody at MIPS who no longer appears to be employed there.

Loading

@mengzhuo
Copy link
Contributor

@mengzhuo mengzhuo commented Sep 18, 2019

@dmitshur Thanks, there are 3 "ok" builds now. It seems good to me.

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Sep 18, 2019

Thanks @mengzhuo for setting up the builder. I'm happy to see MIPS64 port still works.

Loading

@daniel-santos
Copy link

@daniel-santos daniel-santos commented Oct 11, 2019

Hello. I've gotten the OK to make a mipsel board available. These are mt7620-boards, so MIPS 24KEc, with 256MiB of ram. They are not great for building, I cross-compile via OpenWRT and then run the testsuite on the board. I can have them hosted in Fort Lauderdale, Florida. Will this help?

I don't currently have a full toolchain working on the board, but I can probably get one working if you need to build on the board too (it will just be slow).

Loading

@milanknezevic
Copy link
Contributor

@milanknezevic milanknezevic commented Oct 28, 2019

Hello, we would like to dedicate two cavium,rhino_utm8 boards with debian linux for mips and mipsle builders. They have quad-core cpu, 8GB of ram and 240GB ssd disks. Should we start by following these steps https://github.com/golang/go/wiki/DashboardBuilders?

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Oct 28, 2019

@milanknezevic, that sounds great! Thanks.

Would you be the builder's listed owner? Should we name it after you, or your organization? (who is "we", btw?)

Loading

@milanknezevic
Copy link
Contributor

@milanknezevic milanknezevic commented Oct 29, 2019

@bradfitz
Sorry, we are RT-RK, company based in Novi Sad, Serbia. The owner would be @bogojevic, and I would help him setup the builders.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Oct 29, 2019

@milanknezevic, I can set up the config for them after I get a bit more info. It seems that that processor is a 64-bit MIPS64 processor? And it seems that MIPS64 can also run 32-bit MIPS code? Can you confirm? Are you able to run both a cross-compiled GOARCH=mips and GOARCH=mips64 binary on one host and GOARCH=mipsle and GOARCH=mips64le on the other? Are the two boards the same hardware but boot in different endianness modes/kernels?

If so, should I set up each board to test both 32- and 64- bit GOARCHes?

/cc @cherrymui who probably knows how this works.

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Oct 29, 2019

64-bit MIPS processor almost always can run 32-bit MIPS code. I think the kernel can be configured to support either ABI, or both (probably the default).

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Oct 29, 2019

To clarify, I think a 32-bit kernel can only run 32-bit code, a 64-bit kernel can support either ABI or both.

Loading

@milanknezevic
Copy link
Contributor

@milanknezevic milanknezevic commented Oct 29, 2019

@milanknezevic, I can set up the config for them after I get a bit more info. It seems that that processor is a 64-bit MIPS64 processor? And it seems that MIPS64 can also run 32-bit MIPS code? Can you confirm?

Yes, that is correct, processor is MIPS64 and kernel is 64bit, but GOARCH=mips64 has 32-bit rfs, so I'm not sure if 64bit version is going to work, it should, but haven't tried it yet on this board.

Are you able to run both a cross-compiled GOARCH=mips and GOARCH=mips64 binary on one host and GOARCH=mipsle and GOARCH=mips64le on the other?

all.bash scripts are run on both of them successfully but only for GOARCH=mips and GOARCH=mipsle. Didn't try for 64-bit versions.

Are the two boards the same hardware but boot in different endianness modes/kernels?

The boards are the same hardware with different configuration of endianness.

If so, should I set up each board to test both 32- and 64- bit GOARCHes?

/cc @cherrymui who probably knows how this works.

For now, you can setup 32-bit versions, and we will try to run GOARCH=mips64 and GOARCH=mips64le and give you the results. Currently, we can see one problem with GOARCH=mips64, we do not have 64-bit gdb.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Oct 29, 2019

Currently, we can see one problem with GOARCH=mips64, we do not have 64-bit gdb.

If it works at all, that's great. We can configure it to skip the gdb test.

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 29, 2019

Change https://golang.org/cl/204043 mentions this issue: dashboard: add MIPS builders at RT-RK

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Oct 29, 2019

@milanknezevic, I sent https://go-review.googlesource.com/c/build/+/204043 with the builder config for our side. Let me know if that looks okay. Notably, note the GOROOT_BOOTSTRAP value.

Email me (github username at golang.org) and I'll get you keys.

Loading

@milanknezevic
Copy link
Contributor

@milanknezevic milanknezevic commented Oct 30, 2019

I copied golang bootstraps in /usr/local/go-bootstrap.
all.bash on mips64le board is finished successfully, and on mips64 board, which now has gdb, there is a few gdb tests failing and TestLinuxSendfile test is failing too. I will investigate it further.

Also, I think that you misplaced "GOHOSTARCH=mipsle", // avoids an extra make.bash phase, you probably wanted to add it to mips64le builder.

Loading

gopherbot pushed a commit to golang/build that referenced this issue Oct 30, 2019
Updates golang/go#31217

Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533
Reviewed-on: https://go-review.googlesource.com/c/build/+/204043
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 7, 2019

Change https://golang.org/cl/205797 mentions this issue: stage0: add support for MIPS builders to stage0

Loading

gopherbot pushed a commit to golang/build that referenced this issue Nov 7, 2019
Updates golang/go#31217

Change-Id: Icf328cb39086fa4196940345c35969a8b26eafb0
Reviewed-on: https://go-review.googlesource.com/c/build/+/205797
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
These builders haven't existed for a while, and they consume
horizontal space on the build dashboard, pushing more interesting
columns off of the screen at moderate resolutions.

Updates golang/go#31217

Change-Id: Ib7021b699956d01d17e0ea8b213efd07b17d26f7
Reviewed-on: https://go-review.googlesource.com/c/build/+/193017
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
Updates golang/go#31217

Change-Id: I7bda4f3ddf7dc76bdd974726747621f69a7acfa0
Reviewed-on: https://go-review.googlesource.com/c/build/+/191577
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
The connection between linux-mipsle-mengzhuo and the build farm
is not stable and failed the snapshot upload stage, so we have to
skipsnapshot for that.

Updates golang/go#31217

Change-Id: Ic01620ec12254221f93234e8b25749a09f3dbc9c
Reviewed-on: https://go-review.googlesource.com/c/build/+/195977
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
Updates golang/go#31217

Change-Id: I3c947fa1dcd0f1d938ffa4dfe565a35b2ccc3533
Reviewed-on: https://go-review.googlesource.com/c/build/+/204043
Reviewed-by: Bryan C. Mills <bcmills@google.com>
codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
Updates golang/go#31217

Change-Id: Icf328cb39086fa4196940345c35969a8b26eafb0
Reviewed-on: https://go-review.googlesource.com/c/build/+/205797
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Nov 23, 2019

@milanknezevic, @bogojevic, the RTRK MIPS builders are running an old version of the buildlet (they're reporting "Version 21").

You should be using the golang.org/x/build/cmd/buildlet/stage0 binary to launch the buildlet; it will download the latest buildlet binary per run if needed.

We will be deleting some code from the build system soon here that drops support for Version 22 and under. Your builders are reporting Version 21. Please either update to master or start using the stage0 binary.

Thanks!

Loading

@bogojevic
Copy link

@bogojevic bogojevic commented Nov 25, 2019

@bradfitz, I'll take a look. The buildlet binary on both boards are from March 9th. When I manually start stage0 binary, download me the same buildlet again.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Nov 25, 2019

@bogojevic, oh, if you were already using the stage0, perhaps we just had old buildlets uploaded to GCS. I just rebuilt buildlet.linux-mips64 & buildlet.linux-mips64le and they're re-uploaded.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Dec 6, 2019

MIPS all looks happy. Closing.

Loading

@bradfitz bradfitz closed this Dec 6, 2019
@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Dec 6, 2019

(Thanks again, RT-RK!)

Loading

@golang golang locked and limited conversation to collaborators Dec 5, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet