Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve kustomization traversal performance #435

Merged
merged 2 commits into from
Dec 20, 2023
Merged

Conversation

allenporter
Copy link
Owner

@allenporter allenporter commented Dec 20, 2023

Improve traversal performance. Speed up by running kustomizations in parallel again, which is now possible after previous simplifications. Even for a small cluster we see a large impact win:

Old:

# time flux-local get hr --path tests/testdata/cluster
NAME            REVISION    CHART                       SOURCE          
weave-gitops    4.0.22      flux-system-weave-gitops    weave-charts    

real    0m4.229s
user    0m5.476s
sys     0m2.140s

New:

# time flux-local get hr --path tests/testdata/cluster
NAME            REVISION    CHART                       SOURCE          
weave-gitops    4.0.22      flux-system-weave-gitops    weave-charts    

real    0m0.996s
user    0m2.304s
sys     0m0.376s

For my own cluster before

$ time flux-local get hr -A --path clusters/prod/flux-system/
...
real    0m14.109s
user    0m19.698s
sys     0m8.298s

After:

$ time flux-local get hr -A --path clusters/prod/flux-system/
...
real    0m5.313s
user    0m10.584s
sys     0m2.383s

@allenporter allenporter changed the title Improve traversal performance Improve kustomization traversal performance Dec 20, 2023
@codecov-commenter
Copy link

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (c50ba40) 92.68% compared to head (f1fe16a) 92.79%.

Files Patch % Lines
flux_local/git_repo.py 87.03% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #435      +/-   ##
==========================================
+ Coverage   92.68%   92.79%   +0.10%     
==========================================
  Files          18       18              
  Lines        1681     1706      +25     
==========================================
+ Hits         1558     1583      +25     
  Misses        123      123              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@allenporter allenporter merged commit b790bfb into main Dec 20, 2023
26 checks passed
sp3nx0r pushed a commit to sp3nx0r/homelab that referenced this pull request Dec 24, 2023
)

[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [allenporter/flux-local](https://togithub.com/allenporter/flux-local)
| action | major | `3.2.0` -> `4.2.1` |

---

### Release Notes

<details>
<summary>allenporter/flux-local (allenporter/flux-local)</summary>

###
[`v4.2.1`](https://togithub.com/allenporter/flux-local/releases/tag/4.2.1)

[Compare
Source](https://togithub.com/allenporter/flux-local/compare/4.2.0...4.2.1)

#### What's Changed

Version `4.2.0` contains a bug in the diff action that caused it to
never emit a diff, which is fixed in this release.

- Update pages workflow to new versions by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#460
- fix: change running user to 1001 by
[@&#8203;onedr0p](https://togithub.com/onedr0p) in
[allenporter/flux-local#461
- Initial design ideas for flux-local v5 by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#463
- Upgrade to pydantic v2 APIs by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#464
- Build with python 3.12 by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#465
- Fix broken diff action output by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#468

#### Developer updates

- chore(deps): update dependency coverage to v7.3.4 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#469
- chore(deps): update dependency pydantic to v2.5.3 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#470

**Full Changelog**:
allenporter/flux-local@4.2.0...4.2.1

###
[`v4.2.0`](https://togithub.com/allenporter/flux-local/releases/tag/4.2.0)

[Compare
Source](https://togithub.com/allenporter/flux-local/compare/4.1.1...4.2.0)

#### What's Changed

- feat: add container build and workflows by
[@&#8203;onedr0p](https://togithub.com/onedr0p) in
[allenporter/flux-local#452
- fix: update logic in container test to build on main by
[@&#8203;onedr0p](https://togithub.com/onedr0p) in
[allenporter/flux-local#453
- fix: change CMD to ENTRYPOINT in Dockerfile by
[@&#8203;onedr0p](https://togithub.com/onedr0p) in
[allenporter/flux-local#454
- Streamline devcontainer by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#456
- Add --output-file flag to get cluster action by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#457
- Add --output-file for build and diff and use in action by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#459

#### New Contributors

- [@&#8203;onedr0p](https://togithub.com/onedr0p) made their first
contribution in
[allenporter/flux-local#452

**Full Changelog**:
allenporter/flux-local@4.1.1...4.2.0

###
[`v4.1.1`](https://togithub.com/allenporter/flux-local/releases/tag/4.1.1)

[Compare
Source](https://togithub.com/allenporter/flux-local/compare/4.1.0...4.1.1)

#### What's Changed

- Remove complexity from multi-cluster diff by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#447
- Improve timeout error message by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#448
- Update stale helm pydoc by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#449
- Cleanup visitor namespace related logic by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#450
- Fix bug in actions by setting cache dependency paths based on action
by [@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#451

**Full Changelog**:
allenporter/flux-local@4.1.0...4.1.1

###
[`v4.1.0`](https://togithub.com/allenporter/flux-local/releases/tag/4.1.0)

[Compare
Source](https://togithub.com/allenporter/flux-local/compare/4.0.0...4.1.0)

#### What's Changed

- Enable caching of package dependencies by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#442
- Fix formatting by running black on code by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#444
- Add container images from Kustomziations to the cluster manifest by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#443
- Add HelmRelease images to cluster manifest by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#445

#### Developer dependency updates

- Revert "Update actions/upload-pages-artifact action to v3" by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#438
- Revert "Update actions/deploy-pages action to v4" by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#439

**Full Changelog**:
allenporter/flux-local@4.0.0...4.1.0

###
[`v4.0.0`](https://togithub.com/allenporter/flux-local/releases/tag/4.0.0)

[Compare
Source](https://togithub.com/allenporter/flux-local/compare/3.2.0...4.0.0)

#### What's Changed

##### Native flux build

The primary change in this release is to move to use `flux build`
internally. This means that the `flux` CLI is now required to be present
in the environment, similar to how `kustomize` is already required. See
documentation for the [Github
Action](https://togithub.com/allenporter/flux-local?tab=readme-ov-file#github-action)
for details on how to setup the CLI environment.
[allenporter/flux-local#392

To simplify internals, multiple clusters in the same path are no longer
supported. You can still use a multi-cluster repo if you specify
`--path` to one of the clusters. This should be a no-op for most users
of the github action since likely you already were setting a path
argument.

##### Diagnostics

You can now discover malormed or unexpected yaml files with `flux-local
diagnostics`
[allenporter/flux-local#401:

    ### flux-local diagnostics
    [DIAGNOSTICS OK]

##### Performance

With the move to `flux build`, we've been able to reduce some
unnecessary `kustomize` calls, improving internal caching, and
parallelize some of the cluster commands. Speed has been improved by 2x
even for smaller clusters with only a handful of kustomizations.
[allenporter/flux-local#435

#### Change Log

- Add flux dependency in development environment by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#391
- Migrate tests to syrup by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#393
- Add tests that trace the full command stack of building the manifest
by [@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#394
- Increase coverage for cluster3 by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#395
- Improve test coverage for cluster3 cases by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#396
- Improve test coverage for internals of git repo manifest builders by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#397
- Remove support for multiple clusters in a single path by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#398
- Skip unnecessary kustomize cfg step when scanning the cluster by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#399
- Add a `diagnostics` command to find yaml files that kustomize does no…
by [@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#401
- Renovate Dockerfile and versions by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#422
- Use `flux build ks` when building Kustomizations by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#392
- Update documentation examples to include flux cli by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#424
- Improve error messages for shell commands by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#425
- Fix support for Kustomizations with absolute paths by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#426
- Improve logging/tracing and track performance counters by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#427
- Improve kustomization traversal performance by
[@&#8203;allenporter](https://togithub.com/allenporter) in
[allenporter/flux-local#435

#### Developer dependency updates

- Update dependency pydantic to v2.5.2 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#400
- Update dependency mypy to v1.7.1 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#402
- Update dependency wheel to v0.42.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#404
- Update pypa/gh-action-pypi-publish action to v1.8.11 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#405
- Update ubuntu Docker tag to jammy-20231128 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#406
- Update dependency pytest-asyncio to v0.23.2 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#410
- Update dependency ruff to v0.1.7 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#412
- Update pre-commit hook charliermarsh/ruff-pre-commit to v0.1.7 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#413
- Update actions/setup-python action to v5 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#411
- Update actions/deploy-pages action to v3 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#409
- Update actions/configure-pages action to v4 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#408
- Update dependency pre-commit to v3.6.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#414
- Update dependency typing-extensions to v4.9.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#415
- Update dependency black to v23.12.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#416
- Update pre-commit hook psf/black to v23.12.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#417
- Update dependency ruff to v0.1.8 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#418
- Update pre-commit hook charliermarsh/ruff-pre-commit to v0.1.8 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#419
- Update dependency coverage to v7.3.3 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#420
- Update dependency pdoc to v14.2.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#421
- Update dependency pip to v23.3.2 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#428
- Update dependency fluxcd/flux2 to v2.2.1 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#429
- Update dependency helm/helm to v3.13.3 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#432
- Update dependency kyverno/kyverno to v1.11.1 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#433
- Update actions/deploy-pages action to v4 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#430
- Update actions/upload-pages-artifact action to v3 by
[@&#8203;renovate](https://togithub.com/renovate) in
[allenporter/flux-local#431
-

**Full Changelog**:
allenporter/flux-local@3.2.0...4.0.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "on saturday" (UTC), Automerge - At
any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/sp3nx0r/homelab).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xMDMuMSIsInVwZGF0ZWRJblZlciI6IjM3LjEwMy4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants