Mass rebuild of packages on jdk+1
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
base
jdk
results
.gitignore
BLACKLIST
Makefile
README.md
classify.sh
compiler-plugin.patch
notes.md

README.md

Find Debian Java issues

This is a collection of hacky scripts to mass-build packages with some variation set-up in the build environment.

Building

The specific variations here change frequently, but the scripts are pretty resilient, mostly because there's nearly nothing here: There's about 25 lines of Makefile, with only a few weird makeisms, and a couple of Dockerfile statements. That's it.

First, we find some packages. I do this by grepping all-build-deps-amd64.txt, which is generated by dose, and a whole lot of terrible Makefiles/Dockerfiles.

Then, we call make with all the packages we want, with .pkg rammed on the end:

% P=$(<all-build-deps-amd64.txt fgrep default-jdk | cut -d' ' -f 1 | sed 's/$/.pkg/')
make -j8 ${=P}

make goes off and does all the builds, leaving thousands of megabytes of docker container lying around (maybe it should run it with --rm? Ho ho.), and some logs, named:

  • foo.pkg for foo succeeding
  • foo.pkg.fail if foo failed.
  • foo.pkg.wip if foo is still building (or if the machine locked up during a build).

Re-running make will rebuild everything that failed the first time.

I've tried to make make make noise about what it's doing, but there's a problem somewhere: sometimes it will hang silently during one of the container builds. Shrug, don't care.

Variations

One of the patches kicking around in this repo is to hack the JDK to accept older source levels, and silently upgrade them. It's supposed to print to stderr, but I don't think this works in many cases. Also, it doesn't catch the <javadoc source= case that the ant hacks are currently missing.

classify.sh

Also in this repo is a script named classify.sh, which tries to guess the problem by grepping the logs. It's awful, ignore it. It needs:

bzcat reproducible.json.bz2 | jq -r '.[]|select(.architecture=="amd64" and .status != "reproducible" and .status != "unreproducible" and .suite=="unstable")|.package'  > .jenkins
diff -u <(ls -1 *.fail | sed 's/.pkg.fail$//') .jenkins | egrep '^-'  | cut -c2- > .broken

ec2

As it's the future, we can rent big machines and use them to build. Building most of the 1,500 default-jdk deps takes about two hours on an i3.8xlarge, which costs about $3 on the "defined duration" spot market.

I use the Ubuntu Xenial images, 'cos we only really care about Docker and a modern kernel for it. Sorry, Debian!

The notes below are a simplified version of my WIP ec2 tuning document.

Get us a whole load of discs:

#sudo mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=2 /dev/xvdb /dev/xvdc && \
#sudo mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=4 /dev/nvme?n1 && \
sudo mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=8 /dev/nvme?n1 && \
sudo mkfs.xfs -K /dev/md0 && \
sudo mkdir /mnt/data && \ 
sudo mount -o nobarrier /dev/md0 /mnt/data

Install Docker from upstream as the missing flags in the old versions drive me up the wall:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - && \
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" && \
sudo apt update && \
sudo apt install -y docker-ce git wget equivs tmux && \
sudo systemctl stop docker

Move the docker data onto our monstrous partition:

sudo mv /var/lib/docker /mnt/data/ &&
sudo ln -s /mnt/data/docker /var/lib &&
sudo systemctl start docker &&
sudo usermod -a -G docker ubuntu

Grab this repo and the dependency list:

git clone https://github.com/FauxFaux/debjdk && \
cd debjdk && \
wget https://quickdeb.goeswhere.com/default-jdk-dependencies-2018-03-18.txt

Can speed up this stage by copying any built debs over, so you don't have to wait for them to build on your expensive machine.

And, in tmux, cripple the proxy (as deb.debian.org is fine), and go!

tmux
:>base/apt.conf && \
make -j $(nproc) $(cat default-jdk-dependencies-*)

Notes

diff -u .broken <(fgrep -l 'error: unmappable character' $(cat .broken | sed 's/$/.pkg.fail/') | sed 's/.pkg.fail$//') | grep '^-' | cut -c2- | sed 1d > .non-encoding-broken