Automatic building of the binaries in pwnlib/data/binutils

[TethysSvensson]

It would be awesome if we did not depend on the pre-built binaries in pwnlib/data/binutils.

I built them originally a long time ago and I have a hard time remembering how. I am pretty sure that with the exception of the objcopy/objdump it was a fairly standard thing to do.

I think that for those two I did some magic to figure out every possible target to compile them in. Or rather: Almost every possible target, as some of them made binutils now build.

[RobertLarsen]

As in:
if ! test -f pwnlib/data/binutils/something; then
wget http://somewhere/something.tar.gz
...
./configure .... && make && make install
fi
?

[TethysSvensson]

I think that ideally we would want something like that. Perhaps this would also enable us to support more architectures dynamically.

I cannot imagine that we would want to do major changes to binutils, so it does not really make sense to have our own repo.

The major questions I can see are:

• Do we still want support for pre-built binaries?
• If so: How?
• At what point to we want to build those binaries? Install-time? First use of a specific architecture?

[RobertLarsen]

Do we still want support for pre-built binaries?
If by pre-built you mean binaries submitted to GitHub I would like that to go away. It is really convenient and checking out a working copy is fast but how those binaries were built (and even what they really are) is not transparent. A build script while taking time would imho be preferable.

At what point to we want to build those binaries? Install-time? First use of a specific architecture?
I would say at install time, otherwise the code would be cluttered by having to check for the existence of tools.

[zachriggle]

None of this is necessary at all. If I recall correctly, this is all borne by @IdolfHatler wanting _every_ architecture in a objdump.

While that's a cool thing to do, I don't think it has any place in pwntools -- especially as a requirement for common architectures (x86, x86_64, arm, mips, ppc).

With regard to {dis,}assembling for various architectures, most Debians already make that available. I'd recommend changing the current naming structure to match, e.g. from the bundled arm-as to arm-linux-gnueabihf-as or arm-none-eabi-as.

$ apt-cache search binutils | grep '^binutils' | sort
binutils-aarch64-linux-gnu - GNU binary utilities, for aarch64-linux-gnu target
binutils-arm-linux-gnueabi - GNU binary utilities, for arm-linux-gnueabi target
binutils-arm-linux-gnueabihf - GNU binary utilities, for arm-linux-gnueabihf target
binutils-arm-none-eabi - GNU assembler, linker and binary utilities for ARM Cortex-A/R/M processors
binutils-avr - Binary utilities supporting Atmel's AVR targets
binutils-dev - GNU binary utilities (BFD development files)
binutils-doc - Documentation for the GNU assembler, linker and binary utilities
binutils - GNU assembler, linker and binary utilities
binutils-h8300-hms - GNU binary utilities, for h8300-hitachi-coff target
binutils-m68hc1x - binary utilities that support Motorola's 68HC11/12 targets
binutils-mingw-w64 - Cross-binutils for Win32 and Win64 using MinGW-w64
binutils-mingw-w64-i686 - Cross-binutils for Win32 (x86) using MinGW-w64
binutils-mingw-w64-x86-64 - Cross-binutils for Win64 (x64) using MinGW-w64
binutils-msp430 - Binary utilities supporting TI's MSP430 targets
binutils-multiarch - Binary utilities that support multi-arch targets
binutils-multiarch-dev - GNU binary utilities that support multi-arch targets (BFD development files)
binutils-powerpc64le-linux-gnu - GNU binary utilities, for powerpc64le-linux-gnu target
binutils-powerpc-linux-gnu - GNU binary utilities, for powerpc-linux-gnu target
binutils-source - GNU assembler, linker and binary utilities (source)
binutils-static - statically linked binutils tools
binutils-z80 - GNU binary utilities for the z80-unknown-coff target

Do we still want support for pre-built binaries?

No, but I'd vote for distributing a .deb, and also distributing the script to build it, and calling it done. They're easy to build, and we can build with the exact same options/versions as the official package, plus our modifications. Then we just stick them up on Launchpad via ppa. It's important that any additional binaries not be stored in the git repo, as each new revision bloats the total size for a git clone.

Things I would not want to see: Downloading a tarball, huge ./configure lines, and make && sudo make install.

$ sudo apt-get install dpkg-dev
$ sudo apt-get build-dep binutils
$ apt-get source binutils # or binutils=2.24 for specific version
$ cd binutils-*
$ dpkg-buildpackage -us -j8
$ ls -lash ../binutils*deb
3.1M -rw-r--r-- 1 user user 3.1M Sep 10 18:15 ../binutils_2.24-5ubuntu3_amd64.deb
1.9M -rw-r--r-- 1 user user 1.9M Sep 10 18:15 ../binutils-dev_2.24-5ubuntu3_amd64.deb
488K -rw-r--r-- 1 user user 486K Sep 10 18:15 ../binutils-doc_2.24-5ubuntu3_all.deb
1.4M -rw-r--r-- 1 user user 1.4M Sep 10 18:15 ../binutils-multiarch_2.24-5ubuntu3_amd64.deb
4.0K -rw-r--r-- 1 user user 1.2K Sep 10 18:15 ../binutils-multiarch-dev_2.24-5ubuntu3_amd64.deb
 17M -rw-r--r-- 1 user user  17M Sep 10 18:16 ../binutils-source_2.24-5ubuntu3_all.deb
728K -rw-r--r-- 1 user user 726K Sep 10 18:15 ../binutils-static_2.24-5ubuntu3_amd64.deb
680K -rw-r--r-- 1 user user 680K Sep 10 18:15 ../binutils-static-udeb_2.24-5ubuntu3_amd64.udeb

At what point to we want to build those binaries? Install-time? First use of a specific architecture?

I think we should recommend that the users install the existing, Debian/Ubuntu-supplied packages so that we can use e.g. m68hc11-as.

sudo apt-get install binutils-m68hc1x
m68hc11-as --help

If the user does context.arch = 'm68hc11', we should then check to see if the binaries are installed, and make a command suggestion.

[TethysSvensson]

@zachriggle: I disagree that debian is the solution.

We are currently actively supporting both debian, ubuntu and arch linux (and probably loads of other distros, though I do not know of any active users there). A goal of this issue is to support e.g. OpenBSD too.

I do agree about the renaming though. There is no reason not to do that.

[zachriggle]

Ubuntu is a Debian, so that's taken care of (my examples were actually from Ubuntu). Arch has support for deb via dpkg, and also has similar packages (e.g. pacman -S binutils-arm-linux-gnueabihf-bin). OpenBSD also seems to have some packages already available e.g. devel/arm-elf/binutils, but compatibility there would be significantly less.

The .deb approach was for hitting maximum coverage for anyone using pwntools with a single package, if we had to do it somewhere. I still don't think it's a good idea to make a custom binutils a required part of pwntools.

I think that contributing additional cross-builds upstream (e.g. to Debian, Ubuntu, Arch, and OpenBSD) will make all of the integration cleaner, and benefit projects outside of pwntools. If not upstream, then hosting in a manner compatible with the package manager (e.g. PPA for Debian/Ubuntu) is preferable to bundling with pwntools.

[TethysSvensson]

How about a compromise?

Support for using the binaries installed through the package manager and prefer using those. However if they are not available for an architecture that a users tries to use, then a message similar to this is printed:

You were trying to (dis-)assemble on architecture FOO, but you do not have [binary] installed.

You can either install this through your package manager or install a local binutils tools into pwntools
using [some pwntools script for downloading, configure, make and installing inside e.g. ~/.pwntools].

[zachriggle]

Sure, that works.

• Most of pwntools works without anything extra installed
• Users may have the option of the trusted binaries from $distro
• There's also the "Fuck it, I just want this to work. Thank god @IdolfHatler has done the pain-in-the-ass work to get objdump and as working for $obscure_arch."

[TethysSvensson]

So I guess the TODO for this issue is something along the lines of:

• Decide how/where we want our own binaries. Do we want to install them in /usr/local/bin, $HOME/bin, $HOME/.pwntools-binutils or $PWNTOOLS/foobar?
• Decide how we want to supply our own binaries. Pre-built, build-script or both?
• Make sure we are able to use the binaries supplied by the distro if they are present.
• When no objdump/objcopy/as is found, then display an error message with how to get them.
• Remove the pre-built binaries from git.

@zachriggle: Does that sound right to you?

[RobertLarsen]

Sounds wonderful...may I add a possible location?
$HOME/.pwntools/{bin,lib,etc,...}
So that we have one directory for all pwntools needs.

[TethysSvensson]

@RobertLarsen: Well... That is not currently a good fit for how we use the filesystem in pwntools, though you might argue that it should be.

We use the filesystem in the following ways:

• We put mako-templates in $HOME/.pwntools-cache/mako. Unless the .asm files are changed, these are generated only once and take up very little space.
• We put other cache-stuff in /tmp/pwntools-%s-cache. These might be big or become irrelevant over time, so purging them at reboot is desirable.
• We put temporary files in /tmp and delete them again afterwards.
• We put static files we depend on inside the git repository.

We do not at this have anything resembling something you would put inside a lib or an etc directory. I am open to ideas about how we might fit in inside that model, but I have a hard time seeing it.

[RobertLarsen]

In Vegas we were talking about building an elf loader shellcode so that we could more easily build very advanced shellcodes. Working with c2shellcode was mostly a pleasure but there were some caveats so being able to just code shellcodes in C and not worry would be nice. Those I guess would probably be libraries but whether they should go in a lib directory or in some other artifact directory I don't know.
But at least having a $HOME/.pwntools/bin directory would make it possible to also have a $HOME/.pwntools/lib directory in the future if we need it...and other subdirectories.

[zachriggle]

If they're distributed in the normal manner for a given distro (e.g. .deb or .pkg.tar.gz), and based off of the same source and build process used for that distro, I see no reason not to override the system files in /usr/bin, since that's where the package installer would put them, and since they provide all of the functionality provided by the default ones and-then-some. This should also alleviate us of needing to solve all of the dependencies to build the software on various distros (apt-get build-dep for Debian/Ubuntu, makepkg -s for Arch), or worry about distro-specific patches.

Otherwise, I think ~/.pwntools is probably the most standard place to put them.

Regarding a cache directory, for mako or otherwise, it really seems like that ought to be something in /tmp or similar. That said, there's no issue with using ~/.pwntools, it just seems weird.

For additional dependencies that would go in a lib directory, I'd think the system would handle that as these dependencies should also be installed in the ${standard_fashion}.

[TethysSvensson]

Well, the idea with the mako cache is that:

• It's fairly small
• It does not need to be rebuilt unless you change the input files (in which case mako will rebuilt them automatically).

With regards to the binutils binaries, I've more or less come to the opinion that:

• Using the distro is preferred
• It should be easy to install our version
• It should be easy to change your mind and begin using the distro

For this reason I would agree with @RobertLarsen -- ~/.pwntools/bin seems a good a place as any.

If we are to summaries this:

• /tmp is used for files that we prefer to go away by themselves after some time
• ~/.pwntools-cache is for files that we do not see any downside in keeping, but which can be regenerated automatically.
• ~/.pwntools is for files we want to keep permanent, such as objdump and in time maybe configuration files.

@zachriggle, @RobertLarsen: Does that seem reasonable to you?

[TethysSvensson]

So now the question remains, how do we want to create our own binaries?

No matter how we do it, we should recommend that people use their distro, but if for some reason that is not possible (for instance because they want to assemble for $obscure_arch), then how do they get the binaries to do this?

• Do we want a build-script?
• Do we want a build-guide on a github page?
• Do we want prebuilt binaries?

[RobertLarsen]

Does that seem reasonable to you?
I can live with that but would still prefer $HOME/.pwntools/cache just for minimizing the cluttering of $HOME.

[zachriggle]

I agree with @RobertLarsen that ~/.pwntools/cache is preferable. Are the mako files computationally expensive to parse, such that it's actually beneficial to persist them across reboots? I'm still missing why /tmp is bad.

I'll step out of the suggestions binaries bit, since I probably sound like a broken record.

[TethysSvensson]

I just tested how long they take to generate. It is currently about 0.1s to generate the mako templates for all the shellcodes on my box (by running shellcraft). That is not enough to annoy me, but if we got ~5 times more it would be.

I am not really a fan of pwntools needing to "warm up" after every reboot, but if it bothers you, I can live with using /tmp for it.

If not, then I am fine with using ~/.pwntools/cache instead. Or perhaps ~/.cache/pwntools?

@zachriggle: If your opinion simply is "we don't want that", then I do not get it. I get why we do not want it IN pwntools, but what is wrong with having a guide/script of how to get additional binaries?

[zachriggle]

Mako Cache
Nope, doesn't bother me. I just didn't see any big performance difference (~0.1s like you pointed out) on a fresh start after deleting ~/.pwntools-cache. It definitely grows in difference once you add in a lot more shellcode. I could see a 5s delay being annoying after a reboot.

$ for i in {0..100}; do cp -R ~/pwntools/pwnlib/shellcraft/templates/{i386,i386.$i}; done
$ rm -rf ~/.pwntools-cache
$ time shellcraft | wc -l
2280
shellcraft  4.70s user 0.34s system 99% cpu 5.082 total
wc -l  0.00s user 0.00s system 0% cpu 5.082 total
$ time shellcraft | wc -l
2280
shellcraft  0.86s user 0.16s system 99% cpu 1.023 total
wc -l  0.00s user 0.00s system 0% cpu 1.023 total

Binaries
I'm firmly in the 'good-idea' camp. I think we're on the same page and in agreement on what we want to do.

Here's what I'd expect our instructions to look like. Version and architecture agnostic, all dependencies are automatically installed.

What I am/was concerned about is if we tried to do a custom from-source build that looked like wget && ./configure --prefix ~/.pwntools && make && make install. That's liable to break all over the place, be out-of-date (or too-up-to-date), and cause all kinds of get-the-build-dependencies problems.

Cool Note

While I was doing the instructions above, I found that Debian's support via binutils-multiarch is actually better than the custom version of pwntools in certain places (and vice-versa as well). It's only missing aarch64. Here's a diff between the supported targets/arches of the two versions of objdump.

[TethysSvensson]

Mako Cache
Okay, let us put the mako cache in /tmp then! :)

Binaries
This is the basic process you would need to go through when trying to get something to work: http://pwnies.dk/static/flowchart.pdf

We should not necessarily show people that flowchart, but that is how complicated the process is.

Cool Note
I can live with the version in debian. The only thing even of slight interest in our version is the NaCL stuff.

[zachriggle]

Hah, I was agreeing with you on the mako cache thing!

Edit: Ah, I see how you misunderstood. "Nope, doesn't bother me" was talking about which path we use, not the delay. In any case, I literally don't care for the Mako cache, I was just confused as to the why. Your timing and my timing showed the why.

Edit 2: Grr lack of sleep make Zach not words properly.

[zachriggle]

Cool flow chart! The thing that I linked to would slot into the first ??? for Arch and maybe Guide 2.

[TethysSvensson]

It really is a pain that you cannot get a GNU assembler that supports more than one architecture.

Was your guides intended to build multiarch objdump+objcopy or single-arch assemblers?

[zachriggle]

Either for Deb. Just a different build script that apt fetches. No idea
for Arch.

On Friday, September 12, 2014, IdolfHatler notifications@github.com wrote:

It really is a pain that you cannot get a GNU assembler that supports more
than one architecture.

Was your guides intended to build multiarch objdump+objcopy or single-arch
assemblers?

—
Reply to this email directly or view it on GitHub
#105 (comment).

Zach Riggle

[TethysSvensson]

Well... Your build method will not create e.g. an arm assembler on my system.

[zachriggle]

+export DEB_TARGET_ARCH=armhf
+echo $DEB_TARGET_ARCH > debian/target
dpkg-buildpackage -us -j8

$ find debian -iname '*as'
debian/binutils-arm-linux-gnueabihf/usr/bin/arm-linux-gnueabihf-as
debian/binutils-arm-linux-gnueabihf/usr/arm-linux-gnueabihf/bin/as
debian/binutils-arm-linux-gnueabihf/usr/share/doc/binutils-arm-linux-gnueabihf/gas
$ debian/binutils-arm-linux-gnueabihf/usr/bin/arm-linux-gnueabihf-as <(echo nop)
$ debian/binutils-arm-linux-gnueabihf/usr/bin/arm-linux-gnueabihf-objdump -d

a.out:     file format elf32-littlearm


Disassembly of section .text:

00000000 <.text>:
   0:   e1a00000    nop         ; (mov r0, r0)

[zachriggle]

Currently looking at getting the builds to work on Ubuntu Launchpad. They perform the builds on their machines, on source packages that are GPG signed. This would allow us to (effectively) distribute binaries which we can say aren't tampered with, built from source that can be verified to be the same as the official Ubuntu packages.

Currently trying a MIPS target as an example here:
https://launchpad.net/~pwntools/+archive/ubuntu/binutils

Not sure if this'll work properly for cross-arch builds, or how to rename the packages so that we can have multiples. The eventual goal is to have a PPA which will install pwntools via apt-get and follow up with all of the dependencies.

[zachriggle]

This is now completed in an auditable and verifiable way. These packages are built and signed by Ubuntu Launchpad, and are built from the original, unmodified binutils source code.

sudo apt-add-repository ppa:pwntools/binutils
sudo apt-get update
sudo apt-get install binutils-{aarch64,alpha,arm,avr,cris,hppa,ia64,m68k,mips,mips64,msp430,powerpc,powerpc64,s390,sparc,vax,xscale}-linux-gnu

These are automatically detected by the pwnlib.asm used in #158.

$ asm -c cris  'nop'
0f05

[zachriggle]

The script used to upload these packages to Launchpad is below. It uses binutils-powerpc-cross as a starting point, and then just replaces powerpc with the appropriate architecture.

sudo apt-get build-dep binutils gnupg gnupg-agent
sudo apt-get install devscripts binutils-source

export DEBFULLNAME="Zach Riggle"
export DEBEMAIL="zachriggle@gmail.com"
export DIR=$PWD

set -ex

rm -rf binutils-*

for ARCH in aarch64 alpha arm avr cris hppa ia64 m68k mips mips64 msp430 powerpc powerpc64 s390 sparc vax xscale;
do
    cd $DIR
    rm -rf binutils-powerpc-cross-0.10
    apt-get source binutils-powerpc-cross
    cd $DIR/binutils-powerpc-cross-0.10
    sed -i "s|powerpc|$ARCH|ig" debian/control
    sed -i "s|CROSS_ARCH .*|CROSS_ARCH=$ARCH|ig" debian/rules
    sed -i "s|CROSS_GNU_TYPE .*|CROSS_ARCH=$ARCH-linux-gnu|ig" debian/rules

    dch --newversion 0.11pwntools5 --package binutils-$ARCH-cross "Create $ARCH version for pwntools"
    dch --release "Release"
    debuild -S -sa
    # Uncomment for local build
    # dpkg-buildpackage -us -uc -j8
done

cd $DIR

for CHANGES in *source.changes;
do
   dput ppa:pwntools/binutils $CHANGES
done

[zachriggle]

There are now 19 architectures available here: https://launchpad.net/~pwntools/+archive/ubuntu/binutils

If I can get the damn thing to build statically-linked versions, we can just extract the binaries and be ~done with this.

[zachriggle]

Since it's needed by Travis CI, I added more debs for 12.04 / Precise.

[zachriggle]

Pinging @IdolfHatler @br0ns @RobertLarsen

What do we want to do for this? We can do a mixture of the current PPA, and then also give out a build.sh from the @IdolfHatler's one-liner.

Does that satisfy everyone?

[zachriggle]

It's probably not a bad idea to submit a formula to the homebrew project to build these.

[TethysSvensson]

I think we should have instructions both for installing the pre-built ubuntu packages and for building yourself. We should possibly also include pre-built static binaries.

@zachriggle: Could you add instructions to the docs? For the ubuntu version, there should probably also be a few copy-pastable for installing the packages from that ppa instead of just a link.

[zachriggle]

Closing this issue in favor of #272 #273 #274