Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor dockerfiles for simplicity #1786

Merged
merged 47 commits into from
Feb 25, 2021

Conversation

heapcrash
Copy link
Collaborator

@heapcrash heapcrash commented Feb 7, 2021

Important Notes and Background

This PR modifies the Dockerfiles in extra/docker so that we have a non-root user, pwntools, and the "installed" version of pwntools is checked out in /home/pwntools/pwntools.

This makes it easier for other Docker images to build on top of this, as well as paves the way for updating travis/docker to use pwntools/pwntools:stable as a base image which allows local testing to work again.

Notably, pwntools is no longer installed as root, but is installed as an editable install in $HOME/pwntools.

Ideally, we would be able to remove the lines from each of dev / beta / stable that read:

    && python -m pip install --upgrade --user --editable pwntools \
    && python3 -m pip install --upgrade --user --editable pwntools

However, we cannot do this because e.g. dev adds new dependencies on rpyc. This is a small penalty to pay, but overall the Docker images are much more usable for development and downstream usage.

Since Pwntools is no longer installed as root, the base Dockerfile now sets:

PATH="/home/pwntools/.local/bin:${PATH}"

Which should put e.g. pwn into the user path, allowing us to pwn version. Since we install Python2 first, then Python3, the Python3 version prevails. This can be verified from within the Docker image:

$ make -C extra/docker dev &>/dev/null

$ docker run pwntools/pwntools:dev sh -c 'head -1 $(which pwn)'
#!/usr/bin/python3

$ docker run -e PWNLIB_NOTERM=1 pwntools/pwntools:dev pwn version
[*] Pwntools v4.5.0dev-dev-2086a319 (5 days ago)

Less Important Changes

This PR removes some Dockerfiles that are unlikely to have ever been used, because they are broken. These are mostly left around from when the Pwntools base image would inherit from one of these images.

We now pin the base image to Ubuntu Bionic. If / when we decide to upgrade to Focal, we only need to change the base/Dockerfile and fix up any errors. For now, Bionic is fine as it is LTS and still supported.

How to Verify this PR

Since this PR doesn't change any source code, it's probably not easy to validate for e.g. @Arusekk. Here's how you can do it relatively easy:

# Clear any cached images from previous builds, or from Docker Hub
$ docker rmi -f pwntools/pwntools:base pwntools/pwntools:dev

# Must build the base image first
$ make -C extra/docker base

# Build the chosen branch image, based off the "base" image
$ make -C extra/docker dev

# Run a command in the newly-created image
$ docker run -e PWNLIB_NOTERM=1 pwntools/pwntools:dev pwn version                                                                                                                                             
[*] Pwntools v4.5.0dev-dev-2086a319 (5 days ago)

@heapcrash
Copy link
Collaborator Author

I would really like feedback from @maybe-sybr and @Arusekk on this, since it changes a lot of things about how the Dockerfiles are used.

Specifically, with this change, pwntools@dev is only installed for the user pwntools for the pwntools/pwntools:dev Docker image. It is NOT installed systemwide. This likely breaks some use-cases.

If it DOES break use-cases, I can easily just change the name of the images to e.g. dev-live from dev to indicate that the Pwntools installation is live and editable.

@heapcrash
Copy link
Collaborator Author

On follow-up and reconsideration, this is probably not the best way to do this. A separate docker file for development (as opposed to the dev branch) should be added. Doing things this way will break stuff downstream.

@heapcrash heapcrash closed this Feb 7, 2021
@heapcrash
Copy link
Collaborator Author

On re-follow-up, we're already creating the pwntools user. Still seeking feedback from @maybe-sybr and @Arusekk

@heapcrash heapcrash reopened this Feb 7, 2021
@heapcrash heapcrash force-pushed the refactor-dockerfiles-for-simplicity branch from e5d47ca to 00de790 Compare February 7, 2021 09:04
Copy link
Member

@Arusekk Arusekk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I am no expert on containers, this looks good enough. And it looks like there should now be less 'layers' or what they are called.

@heapcrash heapcrash force-pushed the refactor-dockerfiles-for-simplicity branch from 4b92872 to d8b2d11 Compare February 15, 2021 05:48
…e future anyway

This leaves base, stable, beta, and dev as the only Dockerfile images

Simplify extra/Docker so that Pwntools is already checked out, and build precedence works

* Base image always gets built first
* Branch-tracking Dockerfiles now require base to be built
* Use locally-checked-out-repo instead of e.g. git+https://github.com/Gallopsled/pwntools@beta
* Install tested with both Python2 and Python3
* Add Python3 user-script directory to $PATH

Update CHANGELOG with Docker changes

Add develop to Makefile, restore old Dockerfiles

Make development target separate, and automatically launch "docker run -it" with suggested arguments

Add develop Dockerfile
@heapcrash heapcrash force-pushed the refactor-dockerfiles-for-simplicity branch from d8b2d11 to 29e1fdd Compare February 15, 2021 21:34
pwnlib/context/__init__.py Outdated Show resolved Hide resolved
pwnlib/shellcraft/templates/mips/linux/cat.asm Outdated Show resolved Hide resolved
pwnlib/timeout.py Show resolved Hide resolved
pwnlib/tubes/process.py Show resolved Hide resolved
travis/docker/Dockerfile.travis Outdated Show resolved Hide resolved
pwnlib/gdb.py Outdated Show resolved Hide resolved
travis/docker/Dockerfile.travis Outdated Show resolved Hide resolved
travis/docker/Makefile Show resolved Hide resolved
travis/docker/doctest2 Show resolved Hide resolved
travis/docker/tmux.sh Outdated Show resolved Hide resolved
@heapcrash
Copy link
Collaborator Author

This pull request now includes a working Dockerfile at travis/docker/Dockerfile that can be used to run the tests in a local Docker container.

The mechanism to run the tests is:

# Run Python3 doctests
make -C travis/docker doctest3 

# Run Python2 doctests
make -C travis/docker doctest2 

# Spawn an interactive shell in the Docker container
make -C travis/docker shell 

These tests all pass, but this PR needs to be merged in order for Docker Hub to automatically build a ci container that we can use in GitHub Actions to run our doctests.

Next will be the creation of a new GHA workflow that runs the Docker image with a bind-mount so that the Pull Request being tested is the copy of Pwntools being used. make doctestX already set up this bind mount, but we might have to change the path slightly for GHA.

Merging this tonight because:

  • This PR has gotten out of control
  • Tests pass with doctest2 and doctest3
  • The original intent, to update extra/docker, was done a long while ago.

@heapcrash
Copy link
Collaborator Author

CI / test 2.7 fails with only 7 failures, all related to Corefile stuff. This is fewer failures than without this PR, so this is an improvement toward getting GitHub Actions doctests working as well. We need a working CI pipeline for the upcoming release.

Please merge if you agree, @Arusekk

@Arusekk Arusekk merged commit 8a6383a into Gallopsled:dev Feb 25, 2021
@heapcrash heapcrash deleted the refactor-dockerfiles-for-simplicity branch May 31, 2021 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants