Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Operation State API #937

Merged
merged 1 commit into from
May 30, 2024

Conversation

zbirenbaum
Copy link
Member

@zbirenbaum zbirenbaum commented May 29, 2024

Description

Introduces nativelink-state and operation_state_manager API

Checklist

  • Updated documentation if needed
  • Tests added/amended
  • bazel test //... passes locally
  • PR is contained in a single commit, using git amend see some docs

This change is Reviewable

@zbirenbaum zbirenbaum requested a review from allada May 29, 2024 05:10
@CLAassistant
Copy link

CLAassistant commented May 29, 2024

CLA assistant check
All committers have signed the CLA.

@MarcusSorealheis
Copy link
Collaborator

What a massive PR

@zbirenbaum zbirenbaum mentioned this pull request May 29, 2024
@zbirenbaum
Copy link
Member Author

zbirenbaum commented May 29, 2024

What a massive PR

It’s on top of a second one. Blaise wanted me to do it this way and I think its a good idea too. #927 is the bulk of these changes

@zbirenbaum zbirenbaum force-pushed the nativelink-state-api branch 2 times, most recently from 60a9f41 to 91ed36e Compare May 29, 2024 18:58
Copy link
Member

@adam-singer adam-singer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 11 of 11 files at r1, 4 of 5 files at r2, all commit messages.
Reviewable status: 0 of 1 LGTMs obtained, and pending CI: windows-2022 / stable

Copy link
Member

@allada allada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 7 of 11 files at r1, 3 of 5 files at r2.
Reviewable status: 0 of 1 LGTMs obtained, and pending CI: windows-2022 / stable, and 3 discussions need to be resolved


-- commits line 16 at r2:
nit: Expand this.


Cargo.toml line 34 at r2 (raw file):

nativelink-scheduler = { path = "nativelink-scheduler" }
nativelink-service = { path = "nativelink-service" }
nativelink-state = { path = "nativelink-state" }

nit: Don't make this it's own crate, put it under scheduler for now.


nativelink-state/src/operation_state_manager.rs line 43 at r2 (raw file):

#[async_trait]
pub trait ActionStateResult: Send + Sync + 'static {
    async fn as_state(&self) -> Result<Arc<ActionState>, Error>;

nit: Add documentation to these.

@zbirenbaum zbirenbaum force-pushed the nativelink-state-api branch 2 times, most recently from 96d47f5 to 44faf08 Compare May 30, 2024 07:47
Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.
Copy link
Member Author

@zbirenbaum zbirenbaum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 1 LGTMs obtained, and pending CI: Vercel, pre-commit-checks


-- commits line 16 at r2:

Previously, allada (Nathan (Blaise) Bruer) wrote…

nit: Expand this.

Done.


Cargo.toml line 34 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

nit: Don't make this it's own crate, put it under scheduler for now.

Done.


nativelink-state/src/operation_state_manager.rs line 43 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

nit: Add documentation to these.

Done.

Copy link
Member

@allada allada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 8 of 8 files at r4, all commit messages.
Reviewable status: :shipit: complete! 1 of 1 LGTMs obtained

@zbirenbaum zbirenbaum merged commit 1d2d838 into TraceMachina:main May 30, 2024
28 checks passed
nfarah86 pushed a commit to nfarah86/nativelink that referenced this pull request Jun 1, 2024
Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>
MarcusSorealheis pushed a commit to nfarah86/nativelink that referenced this pull request Jun 1, 2024
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216579 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapUMACgkQx/6q6dPS
 ZF2kPQ//ZYXhy26xNq5g2P6U7kf+dpcXok2rzntw9620kdNCLodCm7vceDvwa1mn
 JwVnXdGO7QcrLeSagHk6LewptncYq+hOc3ZA7wUtM3bSUTwPlwJue7zNVwpMSZHD
 9pdu9CDgVWh6h7oR+JoRcyHjJ7NTkSU41aLEtwaHbDWatmtX2Cy4iX0BNk55nWaK
 BtzzuLLgiWrpOXNLL4MWlRdxLbN2YjbxW+2S3Q5eg7SnlERKU+COLoaHYUrk8v9J
 qzK35H05wDiiXHQpnWky895G3DDo/NB1znVpyOf/3sPw+OwUvACoZjORIbeh7+0W
 J5NqfX1ngWTtu0dVT1tehguAplVrBCWGJgJcVs03egVlOVSL/WthdkjW/ZkMfEk5
 pUL9Gje8SyuyEMw3tUc8EV6owwLqqyvVsaO+baVZR5hAbxp/yDaT/fewic6OXa4X
 I58NE5a7w9b5Zsem5kW+yFD5GSa11sybrqffd+ofzPmmDvlVYmJ/CaQvTZ/lQpaA
 IIZLyPYcicC3v2I1+GA//QFPJsUlK0McW+MR7L6zlY395yvS6TTA3EoCBY5KrsYD
 inSfeY2IuMKjV81ihQWd56XiK289GKkvspUmjy84ytke0luqM7Xfpu3cii+gCwoq
 8B0dUCJOqEKg1AJUlM4VOS7vAwc1tJHHrEm05/opIycQxkwNnhs=
 =6j1C
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216536 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapRgACgkQx/6q6dPS
 ZF1U1Q/9G9ZN2PkCUxBc+oTN1BjBbNB/52+xmi+Uao39vrGh2vPIOGYbjXPP36Vv
 NUQ5cckmlsoqC1VWWgJMts27Y6oSlP1JydRlgrXmhnNV/SjcAz6r2woP+3At58hk
 iZN0lqft7lZ1Viycq8bDDEDrT56BiGGbDqO0E3UMZncwp/4s5Cw8M+0DTapIz2PU
 UcxJk2goTlKwPIItlFBUF2dTPL+nxtI95rpIdaGRrP/pn7XaRuY72k5pGMVZHsS+
 OVQBmXqYrSh3nUim+YWagvbDrYJU6lksI4ZDN6cnVgdEsF0xEWguCy3g6tR0WVUQ
 Z5Y9BkedyWt7v1tK+B5cBQgoP2m34pY8TzLFbQTLkv7mrxLIwyNqkDLVJOCPcamk
 dJ+yf5lp54vUkeRnVaTcp9IMEjF0Vjteg3WHTKNK8ZsBs9DnNzb+7HyjDa6bYcau
 e0Y5HVPuC5kcTMK95yTkAabdj1qtBr8QpJ7CErqTRBgbxpcgldd3DTNndHv5/Yjz
 QH/zctjfJcISk/Y+KDTBhZRsNF1VGAzpwsT3KnFUdcTg48xMFC6IBI/YkYywg4Ra
 ZIv6O9h0V7FivbXF5J0KqawpW4lQa6mLDzNwm7ZNODVNb5TEGYu8jsYMzkxaKVPR
 ZGEtFuBOPGgLVYcIbuAFeANBERzhw5KUMe7MBSP2DG7r+50pqQI=
 =zuq5
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216503 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapPcACgkQx/6q6dPS
 ZF0xpQ/+ILuc9NSpJpS6BNw5IUyE4cQsquFZhKBTsX1l8/ZMkDZR6pKdO+gsUdKB
 DyA8OX8VspRAWNoxyiEfLRknwNtclr6XG1znX+WcyVjnaujcezau54dhoEAsImdn
 pgwfN/RpLh3ii1Z/MbpPlDzdDcYkOtDiqF2l4Cr2wQSs72o6qONUmCrycNdipyHa
 lED+VesZOZiZTEBOX56hMGQ/NVSPQ9hisIVjFq1J472v+opU5Ht0cu8KSCRA2lud
 2DCVwpnyjIcG9Bf/XlCMzw9lODbOTqehaGsmES6CSLngjzeojhiHqABc5qYQT6Ip
 umWgSV2/AQsNcMG69WB+uPPT1aBwT//XnBfUoSazhGvFQLn9A7oLPLarhZGUP8M1
 IND49L3OBpzbbdY3ipN9e22f4NcYg7irhy/dzLIHjtBd9CPlUTQTybmRz6KebJom
 6Ha/9CXKPcaW23NqsMNlCoWqSL+cIAEvYgEWcZrceOvkeEHppI6pSLMLyWHw9epF
 Odd3AX8EP18axCNe+GJlCOZlRFq9cNXAYuboa0HpHhBbMP7FiVuQhapR/iWTA1Pv
 2X+hjkWURzpwFz+R/ObTBMiI1YqrTHkD7fukb1ibELenCs8UeIznZ7ScV6YX72Ef
 rOoxoHMrDRbXsWpc3JG3TlEbYpdpiWN09sQQpktCwZtU/uh5JsQ=
 =DmRi
 -----END PGP SIGNATURE-----

[Breaking] Digest function now auto-detected from request (TraceMachina#899)

The digest function the client requested is now properly implemented
in the OriginContext. This allows us to inspect what digest-function
the client requested from anywhere in the code.

If you use `verify` store and use `hash_verification_function`,
simply change it to `"verify_hash": true`. We will now auto-detect
the hash function based on the request and if it is not set, we
use the `default_digest_hash_function` in the `global` config.

Update images for docs (TraceMachina#930)

Update Rust crate proc-macro2 to v1.0.84 (TraceMachina#916)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update dependency protobuf to v26.0.bcr.1 (TraceMachina#887)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate quote to v1.0.36 (TraceMachina#938)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump flake and Bazel modules (TraceMachina#947)

Fixes TraceMachina#943

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update images for docs (TraceMachina#930)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.
MarcusSorealheis pushed a commit to nfarah86/nativelink that referenced this pull request Jun 1, 2024
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216579 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapUMACgkQx/6q6dPS
 ZF2kPQ//ZYXhy26xNq5g2P6U7kf+dpcXok2rzntw9620kdNCLodCm7vceDvwa1mn
 JwVnXdGO7QcrLeSagHk6LewptncYq+hOc3ZA7wUtM3bSUTwPlwJue7zNVwpMSZHD
 9pdu9CDgVWh6h7oR+JoRcyHjJ7NTkSU41aLEtwaHbDWatmtX2Cy4iX0BNk55nWaK
 BtzzuLLgiWrpOXNLL4MWlRdxLbN2YjbxW+2S3Q5eg7SnlERKU+COLoaHYUrk8v9J
 qzK35H05wDiiXHQpnWky895G3DDo/NB1znVpyOf/3sPw+OwUvACoZjORIbeh7+0W
 J5NqfX1ngWTtu0dVT1tehguAplVrBCWGJgJcVs03egVlOVSL/WthdkjW/ZkMfEk5
 pUL9Gje8SyuyEMw3tUc8EV6owwLqqyvVsaO+baVZR5hAbxp/yDaT/fewic6OXa4X
 I58NE5a7w9b5Zsem5kW+yFD5GSa11sybrqffd+ofzPmmDvlVYmJ/CaQvTZ/lQpaA
 IIZLyPYcicC3v2I1+GA//QFPJsUlK0McW+MR7L6zlY395yvS6TTA3EoCBY5KrsYD
 inSfeY2IuMKjV81ihQWd56XiK289GKkvspUmjy84ytke0luqM7Xfpu3cii+gCwoq
 8B0dUCJOqEKg1AJUlM4VOS7vAwc1tJHHrEm05/opIycQxkwNnhs=
 =6j1C
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216536 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapRgACgkQx/6q6dPS
 ZF1U1Q/9G9ZN2PkCUxBc+oTN1BjBbNB/52+xmi+Uao39vrGh2vPIOGYbjXPP36Vv
 NUQ5cckmlsoqC1VWWgJMts27Y6oSlP1JydRlgrXmhnNV/SjcAz6r2woP+3At58hk
 iZN0lqft7lZ1Viycq8bDDEDrT56BiGGbDqO0E3UMZncwp/4s5Cw8M+0DTapIz2PU
 UcxJk2goTlKwPIItlFBUF2dTPL+nxtI95rpIdaGRrP/pn7XaRuY72k5pGMVZHsS+
 OVQBmXqYrSh3nUim+YWagvbDrYJU6lksI4ZDN6cnVgdEsF0xEWguCy3g6tR0WVUQ
 Z5Y9BkedyWt7v1tK+B5cBQgoP2m34pY8TzLFbQTLkv7mrxLIwyNqkDLVJOCPcamk
 dJ+yf5lp54vUkeRnVaTcp9IMEjF0Vjteg3WHTKNK8ZsBs9DnNzb+7HyjDa6bYcau
 e0Y5HVPuC5kcTMK95yTkAabdj1qtBr8QpJ7CErqTRBgbxpcgldd3DTNndHv5/Yjz
 QH/zctjfJcISk/Y+KDTBhZRsNF1VGAzpwsT3KnFUdcTg48xMFC6IBI/YkYywg4Ra
 ZIv6O9h0V7FivbXF5J0KqawpW4lQa6mLDzNwm7ZNODVNb5TEGYu8jsYMzkxaKVPR
 ZGEtFuBOPGgLVYcIbuAFeANBERzhw5KUMe7MBSP2DG7r+50pqQI=
 =zuq5
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216503 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapPcACgkQx/6q6dPS
 ZF0xpQ/+ILuc9NSpJpS6BNw5IUyE4cQsquFZhKBTsX1l8/ZMkDZR6pKdO+gsUdKB
 DyA8OX8VspRAWNoxyiEfLRknwNtclr6XG1znX+WcyVjnaujcezau54dhoEAsImdn
 pgwfN/RpLh3ii1Z/MbpPlDzdDcYkOtDiqF2l4Cr2wQSs72o6qONUmCrycNdipyHa
 lED+VesZOZiZTEBOX56hMGQ/NVSPQ9hisIVjFq1J472v+opU5Ht0cu8KSCRA2lud
 2DCVwpnyjIcG9Bf/XlCMzw9lODbOTqehaGsmES6CSLngjzeojhiHqABc5qYQT6Ip
 umWgSV2/AQsNcMG69WB+uPPT1aBwT//XnBfUoSazhGvFQLn9A7oLPLarhZGUP8M1
 IND49L3OBpzbbdY3ipN9e22f4NcYg7irhy/dzLIHjtBd9CPlUTQTybmRz6KebJom
 6Ha/9CXKPcaW23NqsMNlCoWqSL+cIAEvYgEWcZrceOvkeEHppI6pSLMLyWHw9epF
 Odd3AX8EP18axCNe+GJlCOZlRFq9cNXAYuboa0HpHhBbMP7FiVuQhapR/iWTA1Pv
 2X+hjkWURzpwFz+R/ObTBMiI1YqrTHkD7fukb1ibELenCs8UeIznZ7ScV6YX72Ef
 rOoxoHMrDRbXsWpc3JG3TlEbYpdpiWN09sQQpktCwZtU/uh5JsQ=
 =DmRi
 -----END PGP SIGNATURE-----

[Breaking] Digest function now auto-detected from request (TraceMachina#899)

The digest function the client requested is now properly implemented
in the OriginContext. This allows us to inspect what digest-function
the client requested from anywhere in the code.

If you use `verify` store and use `hash_verification_function`,
simply change it to `"verify_hash": true`. We will now auto-detect
the hash function based on the request and if it is not set, we
use the `default_digest_hash_function` in the `global` config.

Update images for docs (TraceMachina#930)

Update Rust crate proc-macro2 to v1.0.84 (TraceMachina#916)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update dependency protobuf to v26.0.bcr.1 (TraceMachina#887)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate quote to v1.0.36 (TraceMachina#938)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump flake and Bazel modules (TraceMachina#947)

Fixes TraceMachina#943

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update images for docs (TraceMachina#930)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants