Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Digest function now auto-detected from request #899

Merged
merged 1 commit into from
May 18, 2024

Conversation

allada
Copy link
Member

@allada allada commented May 6, 2024

The digest function the client requested is now properly implemented
in the OriginContext. This allows us to inspect what digest-function
the client requested from anywhere in the code.


This change is Reviewable

Copy link
Member Author

@allada allada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+@adam-singer

This is dependent on the previous PR landing first.

Reviewable status: 0 of 1 LGTMs obtained, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Local / ubuntu-22.04, Publish image, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, vale, windows-2022 / stable (waiting on @adam-singer)

Copy link
Member

@adam-singer adam-singer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 11 of 11 files at r1, 12 of 12 files at r2, all commit messages.
Reviewable status: :shipit: complete! 1 of 1 LGTMs obtained

@allada allada force-pushed the add-digest-function-to-context branch 2 times, most recently from 7d93867 to 6c33e74 Compare May 7, 2024 21:02
@allada allada changed the title Add digest function to context Digest function now auto-detected from request May 7, 2024
@allada allada force-pushed the add-digest-function-to-context branch from 6c33e74 to 01a86dd Compare May 7, 2024 21:11
Copy link
Member Author

@allada allada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is the final form.

Local testing shows it working, but still need more larger testing.

I can break this into 1 maybe 2 PRs if you prefer, but most of this is related.

Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Vercel, pre-commit-checks

Copy link
Member

@adam-singer adam-singer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 34 of 34 files at r3, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Remote / large-ubuntu-22.04, and 1 discussions need to be resolved


nativelink-scheduler/tests/cache_lookup_scheduler_test.rs line 119 at r3 (raw file):

        let action_name = ActionInfoHashKey {
            instance_name: "instance".to_string(),
            digest_function: DigestHasherFunc::Sha256.proto_digest_func().try_into()?,

Why does this need to do a try_into when the other passed ones in action_messages_test.rs didnt?

@allada allada force-pushed the add-digest-function-to-context branch from 01a86dd to 465d536 Compare May 8, 2024 16:33
Copy link
Member Author

@allada allada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Analyze (javascript-typescript), Bazel Dev / ubuntu-22.04, Vercel, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, vale, and 1 discussions need to be resolved


nativelink-scheduler/tests/cache_lookup_scheduler_test.rs line 119 at r3 (raw file):

Previously, adam-singer (Adam Singer) wrote…

Why does this need to do a try_into when the other passed ones in action_messages_test.rs didnt?

Copy pasta failure. Fixed.

@allada allada force-pushed the add-digest-function-to-context branch from 465d536 to c6259b9 Compare May 8, 2024 17:09
Copy link
Member

@adam-singer adam-singer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 1 files at r4, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Remote / large-ubuntu-22.04, and 1 discussions need to be resolved


nativelink-scheduler/tests/property_modifier_scheduler_test.rs line 251 at r5 (raw file):

        let action_name = ActionInfoHashKey {
            instance_name: "instance".to_string(),
            digest_function: DigestHasherFunc::Sha256.proto_digest_func().try_into()?,

nit: DigestHasherFunc::Sha256.proto_digest_func().try_into()? -> DigestHasherFunc::Sha256

@allada allada force-pushed the add-digest-function-to-context branch 3 times, most recently from b8de3c3 to 85c190b Compare May 14, 2024 18:09
Copy link
Member

@adam-singer adam-singer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 10 of 10 files at r6, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Remote / large-ubuntu-22.04

@allada allada force-pushed the add-digest-function-to-context branch from 85c190b to 8a47ec3 Compare May 16, 2024 16:01
Copy link
Member Author

@allada allada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / ubuntu-22.04, Publish image, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, vale, windows-2022 / stable


nativelink-scheduler/tests/property_modifier_scheduler_test.rs line 251 at r5 (raw file):

Previously, adam-singer (Adam Singer) wrote…

nit: DigestHasherFunc::Sha256.proto_digest_func().try_into()? -> DigestHasherFunc::Sha256

Done.

@allada allada enabled auto-merge (squash) May 16, 2024 16:01
@allada allada force-pushed the add-digest-function-to-context branch from 8a47ec3 to a3ff0c3 Compare May 16, 2024 19:07
The digest function the client requested is now properly implemented
in the OriginContext. This allows us to inspect what digest-function
the client requested from anywhere in the code.

If you use `verify` store and use `hash_verification_function`,
simply change it to `"verify_hash": true`. We will now auto-detect
the hash function based on the request and if it is not set, we
use the `default_digest_hash_function` in the `global` config.
@allada allada force-pushed the add-digest-function-to-context branch from a3ff0c3 to f927801 Compare May 18, 2024 18:46
@allada allada merged commit 0a33c83 into TraceMachina:main May 18, 2024
28 checks passed
nfarah86 pushed a commit to nfarah86/nativelink that referenced this pull request May 22, 2024
…na#899)

The digest function the client requested is now properly implemented
in the OriginContext. This allows us to inspect what digest-function
the client requested from anywhere in the code.

If you use `verify` store and use `hash_verification_function`,
simply change it to `"verify_hash": true`. We will now auto-detect
the hash function based on the request and if it is not set, we
use the `default_digest_hash_function` in the `global` config.
nfarah86 pushed a commit to nfarah86/nativelink that referenced this pull request May 23, 2024
…na#899)

The digest function the client requested is now properly implemented
in the OriginContext. This allows us to inspect what digest-function
the client requested from anywhere in the code.

If you use `verify` store and use `hash_verification_function`,
simply change it to `"verify_hash": true`. We will now auto-detect
the hash function based on the request and if it is not set, we
use the `default_digest_hash_function` in the `global` config.
MarcusSorealheis pushed a commit to nfarah86/nativelink that referenced this pull request Jun 1, 2024
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216579 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapUMACgkQx/6q6dPS
 ZF2kPQ//ZYXhy26xNq5g2P6U7kf+dpcXok2rzntw9620kdNCLodCm7vceDvwa1mn
 JwVnXdGO7QcrLeSagHk6LewptncYq+hOc3ZA7wUtM3bSUTwPlwJue7zNVwpMSZHD
 9pdu9CDgVWh6h7oR+JoRcyHjJ7NTkSU41aLEtwaHbDWatmtX2Cy4iX0BNk55nWaK
 BtzzuLLgiWrpOXNLL4MWlRdxLbN2YjbxW+2S3Q5eg7SnlERKU+COLoaHYUrk8v9J
 qzK35H05wDiiXHQpnWky895G3DDo/NB1znVpyOf/3sPw+OwUvACoZjORIbeh7+0W
 J5NqfX1ngWTtu0dVT1tehguAplVrBCWGJgJcVs03egVlOVSL/WthdkjW/ZkMfEk5
 pUL9Gje8SyuyEMw3tUc8EV6owwLqqyvVsaO+baVZR5hAbxp/yDaT/fewic6OXa4X
 I58NE5a7w9b5Zsem5kW+yFD5GSa11sybrqffd+ofzPmmDvlVYmJ/CaQvTZ/lQpaA
 IIZLyPYcicC3v2I1+GA//QFPJsUlK0McW+MR7L6zlY395yvS6TTA3EoCBY5KrsYD
 inSfeY2IuMKjV81ihQWd56XiK289GKkvspUmjy84ytke0luqM7Xfpu3cii+gCwoq
 8B0dUCJOqEKg1AJUlM4VOS7vAwc1tJHHrEm05/opIycQxkwNnhs=
 =6j1C
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216536 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapRgACgkQx/6q6dPS
 ZF1U1Q/9G9ZN2PkCUxBc+oTN1BjBbNB/52+xmi+Uao39vrGh2vPIOGYbjXPP36Vv
 NUQ5cckmlsoqC1VWWgJMts27Y6oSlP1JydRlgrXmhnNV/SjcAz6r2woP+3At58hk
 iZN0lqft7lZ1Viycq8bDDEDrT56BiGGbDqO0E3UMZncwp/4s5Cw8M+0DTapIz2PU
 UcxJk2goTlKwPIItlFBUF2dTPL+nxtI95rpIdaGRrP/pn7XaRuY72k5pGMVZHsS+
 OVQBmXqYrSh3nUim+YWagvbDrYJU6lksI4ZDN6cnVgdEsF0xEWguCy3g6tR0WVUQ
 Z5Y9BkedyWt7v1tK+B5cBQgoP2m34pY8TzLFbQTLkv7mrxLIwyNqkDLVJOCPcamk
 dJ+yf5lp54vUkeRnVaTcp9IMEjF0Vjteg3WHTKNK8ZsBs9DnNzb+7HyjDa6bYcau
 e0Y5HVPuC5kcTMK95yTkAabdj1qtBr8QpJ7CErqTRBgbxpcgldd3DTNndHv5/Yjz
 QH/zctjfJcISk/Y+KDTBhZRsNF1VGAzpwsT3KnFUdcTg48xMFC6IBI/YkYywg4Ra
 ZIv6O9h0V7FivbXF5J0KqawpW4lQa6mLDzNwm7ZNODVNb5TEGYu8jsYMzkxaKVPR
 ZGEtFuBOPGgLVYcIbuAFeANBERzhw5KUMe7MBSP2DG7r+50pqQI=
 =zuq5
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216503 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapPcACgkQx/6q6dPS
 ZF0xpQ/+ILuc9NSpJpS6BNw5IUyE4cQsquFZhKBTsX1l8/ZMkDZR6pKdO+gsUdKB
 DyA8OX8VspRAWNoxyiEfLRknwNtclr6XG1znX+WcyVjnaujcezau54dhoEAsImdn
 pgwfN/RpLh3ii1Z/MbpPlDzdDcYkOtDiqF2l4Cr2wQSs72o6qONUmCrycNdipyHa
 lED+VesZOZiZTEBOX56hMGQ/NVSPQ9hisIVjFq1J472v+opU5Ht0cu8KSCRA2lud
 2DCVwpnyjIcG9Bf/XlCMzw9lODbOTqehaGsmES6CSLngjzeojhiHqABc5qYQT6Ip
 umWgSV2/AQsNcMG69WB+uPPT1aBwT//XnBfUoSazhGvFQLn9A7oLPLarhZGUP8M1
 IND49L3OBpzbbdY3ipN9e22f4NcYg7irhy/dzLIHjtBd9CPlUTQTybmRz6KebJom
 6Ha/9CXKPcaW23NqsMNlCoWqSL+cIAEvYgEWcZrceOvkeEHppI6pSLMLyWHw9epF
 Odd3AX8EP18axCNe+GJlCOZlRFq9cNXAYuboa0HpHhBbMP7FiVuQhapR/iWTA1Pv
 2X+hjkWURzpwFz+R/ObTBMiI1YqrTHkD7fukb1ibELenCs8UeIznZ7ScV6YX72Ef
 rOoxoHMrDRbXsWpc3JG3TlEbYpdpiWN09sQQpktCwZtU/uh5JsQ=
 =DmRi
 -----END PGP SIGNATURE-----

[Breaking] Digest function now auto-detected from request (TraceMachina#899)

The digest function the client requested is now properly implemented
in the OriginContext. This allows us to inspect what digest-function
the client requested from anywhere in the code.

If you use `verify` store and use `hash_verification_function`,
simply change it to `"verify_hash": true`. We will now auto-detect
the hash function based on the request and if it is not set, we
use the `default_digest_hash_function` in the `global` config.

Update images for docs (TraceMachina#930)

Update Rust crate proc-macro2 to v1.0.84 (TraceMachina#916)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update dependency protobuf to v26.0.bcr.1 (TraceMachina#887)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate quote to v1.0.36 (TraceMachina#938)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump flake and Bazel modules (TraceMachina#947)

Fixes TraceMachina#943

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update images for docs (TraceMachina#930)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.
MarcusSorealheis pushed a commit to nfarah86/nativelink that referenced this pull request Jun 1, 2024
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216579 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapUMACgkQx/6q6dPS
 ZF2kPQ//ZYXhy26xNq5g2P6U7kf+dpcXok2rzntw9620kdNCLodCm7vceDvwa1mn
 JwVnXdGO7QcrLeSagHk6LewptncYq+hOc3ZA7wUtM3bSUTwPlwJue7zNVwpMSZHD
 9pdu9CDgVWh6h7oR+JoRcyHjJ7NTkSU41aLEtwaHbDWatmtX2Cy4iX0BNk55nWaK
 BtzzuLLgiWrpOXNLL4MWlRdxLbN2YjbxW+2S3Q5eg7SnlERKU+COLoaHYUrk8v9J
 qzK35H05wDiiXHQpnWky895G3DDo/NB1znVpyOf/3sPw+OwUvACoZjORIbeh7+0W
 J5NqfX1ngWTtu0dVT1tehguAplVrBCWGJgJcVs03egVlOVSL/WthdkjW/ZkMfEk5
 pUL9Gje8SyuyEMw3tUc8EV6owwLqqyvVsaO+baVZR5hAbxp/yDaT/fewic6OXa4X
 I58NE5a7w9b5Zsem5kW+yFD5GSa11sybrqffd+ofzPmmDvlVYmJ/CaQvTZ/lQpaA
 IIZLyPYcicC3v2I1+GA//QFPJsUlK0McW+MR7L6zlY395yvS6TTA3EoCBY5KrsYD
 inSfeY2IuMKjV81ihQWd56XiK289GKkvspUmjy84ytke0luqM7Xfpu3cii+gCwoq
 8B0dUCJOqEKg1AJUlM4VOS7vAwc1tJHHrEm05/opIycQxkwNnhs=
 =6j1C
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216536 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapRgACgkQx/6q6dPS
 ZF1U1Q/9G9ZN2PkCUxBc+oTN1BjBbNB/52+xmi+Uao39vrGh2vPIOGYbjXPP36Vv
 NUQ5cckmlsoqC1VWWgJMts27Y6oSlP1JydRlgrXmhnNV/SjcAz6r2woP+3At58hk
 iZN0lqft7lZ1Viycq8bDDEDrT56BiGGbDqO0E3UMZncwp/4s5Cw8M+0DTapIz2PU
 UcxJk2goTlKwPIItlFBUF2dTPL+nxtI95rpIdaGRrP/pn7XaRuY72k5pGMVZHsS+
 OVQBmXqYrSh3nUim+YWagvbDrYJU6lksI4ZDN6cnVgdEsF0xEWguCy3g6tR0WVUQ
 Z5Y9BkedyWt7v1tK+B5cBQgoP2m34pY8TzLFbQTLkv7mrxLIwyNqkDLVJOCPcamk
 dJ+yf5lp54vUkeRnVaTcp9IMEjF0Vjteg3WHTKNK8ZsBs9DnNzb+7HyjDa6bYcau
 e0Y5HVPuC5kcTMK95yTkAabdj1qtBr8QpJ7CErqTRBgbxpcgldd3DTNndHv5/Yjz
 QH/zctjfJcISk/Y+KDTBhZRsNF1VGAzpwsT3KnFUdcTg48xMFC6IBI/YkYywg4Ra
 ZIv6O9h0V7FivbXF5J0KqawpW4lQa6mLDzNwm7ZNODVNb5TEGYu8jsYMzkxaKVPR
 ZGEtFuBOPGgLVYcIbuAFeANBERzhw5KUMe7MBSP2DG7r+50pqQI=
 =zuq5
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700
committer Marcus <marcuseagan@gmail.com> 1717216503 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapPcACgkQx/6q6dPS
 ZF0xpQ/+ILuc9NSpJpS6BNw5IUyE4cQsquFZhKBTsX1l8/ZMkDZR6pKdO+gsUdKB
 DyA8OX8VspRAWNoxyiEfLRknwNtclr6XG1znX+WcyVjnaujcezau54dhoEAsImdn
 pgwfN/RpLh3ii1Z/MbpPlDzdDcYkOtDiqF2l4Cr2wQSs72o6qONUmCrycNdipyHa
 lED+VesZOZiZTEBOX56hMGQ/NVSPQ9hisIVjFq1J472v+opU5Ht0cu8KSCRA2lud
 2DCVwpnyjIcG9Bf/XlCMzw9lODbOTqehaGsmES6CSLngjzeojhiHqABc5qYQT6Ip
 umWgSV2/AQsNcMG69WB+uPPT1aBwT//XnBfUoSazhGvFQLn9A7oLPLarhZGUP8M1
 IND49L3OBpzbbdY3ipN9e22f4NcYg7irhy/dzLIHjtBd9CPlUTQTybmRz6KebJom
 6Ha/9CXKPcaW23NqsMNlCoWqSL+cIAEvYgEWcZrceOvkeEHppI6pSLMLyWHw9epF
 Odd3AX8EP18axCNe+GJlCOZlRFq9cNXAYuboa0HpHhBbMP7FiVuQhapR/iWTA1Pv
 2X+hjkWURzpwFz+R/ObTBMiI1YqrTHkD7fukb1ibELenCs8UeIznZ7ScV6YX72Ef
 rOoxoHMrDRbXsWpc3JG3TlEbYpdpiWN09sQQpktCwZtU/uh5JsQ=
 =DmRi
 -----END PGP SIGNATURE-----

[Breaking] Digest function now auto-detected from request (TraceMachina#899)

The digest function the client requested is now properly implemented
in the OriginContext. This allows us to inspect what digest-function
the client requested from anywhere in the code.

If you use `verify` store and use `hash_verification_function`,
simply change it to `"verify_hash": true`. We will now auto-detect
the hash function based on the request and if it is not set, we
use the `default_digest_hash_function` in the `global` config.

Update images for docs (TraceMachina#930)

Update Rust crate proc-macro2 to v1.0.84 (TraceMachina#916)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update dependency protobuf to v26.0.bcr.1 (TraceMachina#887)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate quote to v1.0.36 (TraceMachina#938)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump flake and Bazel modules (TraceMachina#947)

Fixes TraceMachina#943

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update images for docs (TraceMachina#930)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <zach@tracemachina.com>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants