Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add buildah transport #595

Merged
merged 2 commits into from Jun 13, 2019

Conversation

@jordemort
Copy link
Contributor

commented Jun 9, 2019

This adds a transport for Buildah - with this and a little bit of hacking, I am able to accelerate ansible-bender with Mitogen. This brings my container build time down from 12 minutes to 5 minutes.

TODO:

  • Add entry to docs/changelog.rst
  • Add stub and test
@@ -0,0 +1,73 @@
# Copyright 2019, David Wilson

This comment has been minimized.

Copy link
@jordemort

jordemort Jun 9, 2019

Author Contributor

This is literally just a slightly modified version of the Docker plugin so I just left your copyright.

@jordemort

This comment has been minimized.

Copy link
Contributor Author

commented Jun 9, 2019

Note that current versions of buildah have a bug in handling stdin on buildah run and you may need a version of buildah with containers/buildah#1653 applied if you want to actually try this out.

@jordemort jordemort marked this pull request as ready for review Jun 10, 2019

@jordemort

This comment has been minimized.

Copy link
Contributor Author

commented Jun 12, 2019

Not sure what's up with CI here, failure looks unrelated.

@dw

This comment has been minimized.

Copy link
Owner

commented Jun 13, 2019

Sorry for the delay, I have e-mails for the project getting archived just now :)

The CI failures are probably the Python 2.4 job which is missing a fork safety hook in one place, or the Ansible jobs which fail due to a race in the su method which is fixed on the async branch. Sorry about that

This looks perfect, thanks a ton!

@dw dw merged commit 4fedf88 into dw:master Jun 13, 2019

1 of 2 checks passed

dw.mitogen #20190610.1 failed
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@dw

This comment has been minimized.

Copy link
Owner

commented Jun 13, 2019

Regarding the stdin issue, I'll try things out locally. There are quite a few knobs available to work around dodgy tooling, it's possible we already have a workaround for the problem in Buildah, but if not, we could always add one if it looks straightforward

@jordemort

This comment has been minimized.

Copy link
Contributor Author

commented Jun 13, 2019

@dw The PR to resolve my stdin handling issue (and a subsequent one that resolved even more) have been merged to buildah master (though they don't look like they have, since they don't actually use the GitHub merge function in their workflow) and should be in their 1.8.4 release.

dw added a commit that referenced this pull request Aug 15, 2019
dw added a commit that referenced this pull request Aug 15, 2019
dw added a commit that referenced this pull request Aug 17, 2019
Merge remote-tracking branch 'origin/dmw'
* origin/dmw:
  issue #615: ensure 4GB max_message_size is configured for task workers.
  issue #615: update Changelog.
  issue #615: route a dead message to recipients when no reply is expected
  issue #615: fetch_file() might be called with AnsibleUnicode.
  issue #615: redirect 'fetch' action to 'mitogen_fetch'.
  issue #615: extricate slurp brainwrong from mitogen_fetch
  issue #615: ansible: import Ansible fetch.py action plug-in
  issue #533: include object identity of Stream in repr()
  docs: lots more changelog
  issue #595: add buildah to docs and changelog.
  docs: a few more internals.rst additions
dw added a commit that referenced this pull request Aug 18, 2019
Merge remote-tracking branch 'origin/v028' into stable
* origin/v028: (383 commits)
  Bump version for release.
  docs: update Changelog for 0.2.8.
  issue #627: add test and tweak Reaper behaviour.
  docs: lots more changelog concision
  docs: changelog concision
  docs: more changelog tweaks
  docs: reorder chapters
  docs: versionless <title>
  docs: update supported Ansible version, mention unsupported features
  docs: changelog fixes/tweaks
  issue #590: update Changelog.
  issue #621: send ADD_ROUTE earlier and add test for early logging.
  issue #590: whoops, import missing test modules
  issue #590: rework ParentEnumerationMethod to recursively handle bad modules
  issue #627: reduce the default pool size in a child to 2.
  tests: add a few extra service tests.
  docs: some more hyperlink joy
  docs: more hyperlinks
  docs: add domainrefs plugin to make link aliases everywhere \o/
  docs: link IS_DEAD in changelog
  docs: tweaks to better explain changelog race
  issue #533: update routing to account for DEL_ROUTE propagation race
  tests: use defer_sync() Rather than defer() + ancient sync_with_broker()
  tests: one case from doas_test was invoking su
  tests: hide memory-mapped files from lsof output
  issue #615: remove meaningless test
  issue #625: ignore SIGINT within MuxProcess
  issue #625: use exec() instead of subprocess in mitogen_ansible_playbook
  issue #615: regression test
  issue #615: update Changelog.
  issue #615: ensure 4GB max_message_size is configured for task workers.
  issue #615: update Changelog.
  issue #615: route a dead message to recipients when no reply is expected
  issue #615: fetch_file() might be called with AnsibleUnicode.
  issue #615: redirect 'fetch' action to 'mitogen_fetch'.
  issue #615: extricate slurp brainwrong from mitogen_fetch
  issue #615: ansible: import Ansible fetch.py action plug-in
  issue #533: include object identity of Stream in repr()
  docs: lots more changelog
  issue #595: add buildah to docs and changelog.
  docs: a few more internals.rst additions
  ci: update to Ansible 2.8.3
  tests: another random string changed in 2.8.3
  tests: fix sudo_flags_failure for Ansible 2.8.3
  ci: fix procps command line format warning
  Whoops, merge together lgtm.yml and .lgtm.yml
  issue #440: log Python version during bootstrap.
  docs: update changelog
  issue #558: disable test on OSX to cope with boundless mediocrity
  issue #558, #582: preserve remote tmpdir if caller did not supply one
  issue #613: must await 'exit' and 'disconnect' in wait=False test
  Import LGTM config to disable some stuff
  Fix up another handful of LGTM errors.
  tests: work around AnsibleModule.run_command() race.
  docs: mention another __main__ safeguard
  docs: tweaks
  formatting error
  docs: make Sphinx install soft fail on Python 2.
  issue #598: allow disabling preempt in terraform
  issue #598: update Changelog.
  issue #605: update Changelog.
  issue #605: ansible: share a sem_t instead of a pthread_mutex_t
  issue #613: add tests for all the weird shutdown methods
  Add mitogen.core.now() and use it everywhere; closes #614.
  docs: move decorator docs into core.py and use autodecorator
  preamble_size: make it work on Python 3.
  docs: upgrade Sphinx to 2.1.2, require Python 3 to build docs.
  docs: fix Sphinx warnings, add LogHandler, more docstrings
  docs: tidy up some Changelog text
  issue #615: fix up FileService tests for new logic
  issue #615: another Py3x fix.
  issue #615: Py3x fix.
  issue #615: update Changelog.
  issue #615: use FileService for target->controll file transfers
  issue #482: another Py3 fix
  ci: try removing exclude: to make Azure jobs work again
  compat: fix Py2.4 SyntaxError
  issue #482: remove 'ssh' from checked processes
  ci: Py3 fix
  issue #279: add one more test for max_message_size
  issue #482: ci: add stray process checks to all jobs
  tests: fix format string error
  core: MitogenProtocol.is_privileged was not set in children
  issue #482: tests: fail DockerMixin tests if stray processes exist
  docs: update Changelog.
  issue #586: update Changelog.
  docs: update Changelog.
  [security] core: undirectional routing wasn't respected in some cases
  docs: tidy up Select.all()
  issue #612: update Changelog.
  master: fix TypeError
  pkgutil: fix Python3 compatibility
  parent: use protocol for getting remote_id
  docs: merge signals.rst into internals.rst
  os_fork: do not attempt to cork the active thread.
  parent: fix get_log_level() for split out loggers.
  issue #547: fix service_test failures.
  issue #547: update Changelog.
  issue #547: core/service: race/deadlock-free service pool init
  docs: update Changelog.
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.