WIP: deploy: Attempt accelerated deploy by updating old deploy #1419
Conversation
wmanley
force-pushed
the
reuse-deployment-dirs
branch
4 times, most recently
from
e588cba
to
c45f5ba
Compare
|
☔ The latest upstream changes (presumably 8174885) made this pull request unmergeable. Please resolve the merge conflicts. |
Hah, that's quite close to the number of files on this desktop: |
|
I just want to note I haven't forgotten about your PRs here; I feel a bit guilty every time I see the PR list as I really want to get them in. Currently the staging bits (as part of auto-updates) have stayed at the top of my priority queue for a while, and these patches also really want a better testing framework than we have today, which is also slowly happening. As the new ansible/VM framework lands and improves we can more easily test these types of patches. |
This is in preparation for adding `OSTREE_REPO_CHECKOUT_OVERWRITE_UPDATE` where we will need to iterate through two directories simultaneously.
This is in preparation for adding `OSTREE_REPO_CHECKOUT_OVERWRITE_UPDATE` where we will need to iterate through two directories simultaneously.
Out of date since e6f17b9.
Used for updating an already checked out tree to a new version. This is intended to make deploys much faster in the future. In default operation `checkout_tree_at` now is doing an update from an empty tree to the given one. This implements diffs for directories. The next commit will introduce the same for files.
The goal is to do ostree deploys faster by updating already checked out trees. This commit continues the work in the previous commit. `checkout_tree_at` now takes an `orig_dirtree_checksum` telling it to assume that the directory it's checking out into already has this content.
The goal is to do ostree deploys faster by updating already checked out trees. This adds an overwrite mode and a new option to specify the original tree. It will do a lot less work when you're updating a tree with a large number of files but only a few files have changed. So if you have a full distro tree checked out and want to update `/usr/bin/ostree` we no longer have to recurse through every file in `/usr/share/doc` to do the update.
This exposes the OSTREE_REPO_CHECKOUT_OVERWRITE_UPDATE functionality added in the previous commit on the command line.
To avoid having any incompletely checked-out or corrupted trees as
`/ostree/deploy/OS/deploy/${treecsum}.${deployserial}`. We'll need this
so we can trust ${treecsum} for updating existing unused deployments.
...rather than doing a complete fresh checkout. Currently on our (embedded) system with slow eMMC disk it takes 1m40s to do a deploy (~150000 files). Ideally if only one or two files have changed it should be nearly instantaneous - suitable for interactive development. This should also reduce eMMC wear. On any failure we fall back to doing a full deploy.
to enable detailed logging of checkout operations.
so the ordering in the log files between the `sh -x` output and the diff output is preserved. I was getting very confused debugging an issue because the diff was mixed in with the shell command lines which made it look like there was no diff.
wmanley
force-pushed
the
reuse-deployment-dirs
branch
from
11c87bf
to
1c964f7
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: wmanley The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Rebased on master. |
|
@wmanley: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@wmanley: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
...rather than doing a complete fresh checkout.
Currently on our (embedded) system with slow eMMC disk it takes 1m40s to do
a deploy (~150000 files). Ideally if only one or two files have changed it
should be nearly instantaneous - suitable for interactive development. This
should also reduce eMMC wear.
After applying these patches (and with
--no-prune#1418) it takes <2s to do a deploy - a 50x-100x speedup.We store unpacked container images as part of our rootfs. Checkouts with O(number of changes) rather than O(number of files) is going to be more and more important for us as we add more containers.
On any failure we fall back to doing a full deploy.
This builds on top of #1408. Although it's not ready I've raised this PR to solicit opinion.
TODO:
/usrfrom old deploys - rather than removing/etcsyncfsbefore moving deployment into final directory.