JobResult Table Re-design - Improved task- and machine- granularity #3539

nharper285 · 2023-10-02T21:52:35Z

Summary of the Pull Request

What is this about?

PR Checklist

Applies to work item: RuntimeStats are not agregated per nodes #3512
CLA signed. If not, go over here and sign the CLI.
Tests added/passed
Requires documentation to be updated
I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

Info on Pull Request

What does this include?

Validation Steps Performed

How does someone test & validate?

* Remove the retention policy setting (microsoft#3452) --------- Co-authored-by: Cheick Keita <chkeita@microsoft.com>

This reverts commit c69deed.

* Redo-8.7.1-hotfix --------- Co-authored-by: Cheick Keita <chkeita@microsoft.com>

…t#3467) * Add field to ado config for checking duplicate work items * Make duplicate fields nullable and add it to python models * Update broken tests * Update docs to include new ado_duplicate_fields property

Co-authored-by: Adam <103067949+AdamL-Microsoft@users.noreply.github.com>

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.30.0 to 1.32.0. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](tokio-rs/tokio@tokio-1.30.0...tokio-1.32.0) --- updated-dependencies: - dependency-name: tokio dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [elsa](https://github.com/manishearth/elsa) from 1.8.1 to 1.9.0. - [Commits](Manishearth/elsa@v1.8.1...v1.9.0) --- updated-dependencies: - dependency-name: elsa dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.7.1 to 3.8.0. - [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md) - [Commits](Stebalien/tempfile@v3.7.1...v3.8.0) --- updated-dependencies: - dependency-name: tempfile dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Updating requirements.txt to accept >= onefuzztypes. * Trying to loosen restriction.

Bumps [notify](https://github.com/notify-rs/notify) from 6.0.1 to 6.1.1. - [Release notes](https://github.com/notify-rs/notify/releases) - [Changelog](https://github.com/notify-rs/notify/blob/main/CHANGELOG.md) - [Commits](notify-rs/notify@notify-6.0.1...notify-6.1.1) --- updated-dependencies: - dependency-name: notify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Release 8.8.0

Bumps [clap](https://github.com/clap-rs/clap) from 4.3.21 to 4.4.2. - [Release notes](https://github.com/clap-rs/clap/releases) - [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md) - [Commits](clap-rs/clap@v4.3.21...v4.4.2) --- updated-dependencies: - dependency-name: clap dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [gimli](https://github.com/gimli-rs/gimli) from 0.27.3 to 0.28.0. - [Changelog](https://github.com/gimli-rs/gimli/blob/master/CHANGELOG.md) - [Commits](gimli-rs/gimli@0.27.3...0.28.0) --- updated-dependencies: - dependency-name: gimli dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [clap](https://github.com/clap-rs/clap) from 4.3.21 to 4.4.2. - [Release notes](https://github.com/clap-rs/clap/releases) - [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md) - [Commits](clap-rs/clap@v4.3.21...v4.4.2) --- updated-dependencies: - dependency-name: clap dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [winreg](https://github.com/gentoo90/winreg-rs) from 0.50.0 to 0.51.0. - [Release notes](https://github.com/gentoo90/winreg-rs/releases) - [Changelog](https://github.com/gentoo90/winreg-rs/blob/master/CHANGELOG.md) - [Commits](gentoo90/winreg-rs@v0.50.0...v0.51.0) --- updated-dependencies: - dependency-name: winreg dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adam <103067949+AdamL-Microsoft@users.noreply.github.com>

* Starting integration tests * Ready to test the test * Parametrize test * checkpoint * Test works * Run integration tests in pipeline * fmt * . * -p * Install clang * quotes not required in yaml? * Hopefully fixed windows? * Try without killondrop * lint * small test * another test * Reuse core name * Wrong step * bump tokio? * Try with rust * make build happy * Bump pete and small clean up * Clean up and make the test pass regularly * fix broken ci * Lower the poll timeout * Set the timeout in a nicer way * fix windows * fmt * Include and copy pdbs * Ignore if pdb is missing on linux * It takes too long for coverage to be generated * lint * Only warn on missing coverage since it's flaky * Fix windows build * Small clean up * Try lowering the poll delay * fix coverage * PR comments * . * Apparently make is missing? * Remove aggressive step skipping in CI

* Fix sed checks for CLI versioning * Fix. * Fix. * Changing build_cli * Trying greater than * Tring once more. * Trying major minor * trying to replace major minor * Using major minor

Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](tokio-rs/bytes@v1.4.0...v1.5.0) --- updated-dependencies: - dependency-name: bytes dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add more comprehensive checks and better error messages to area/iteration path validation * Join invalid chars with space instead of comma * Make tree path validation more testable * Add error code for invalid ADO project in config * Write unit tests for tree path validation * Format tree path unit tests * Merge escape character and control character checks and clarify error message

* Improve handling of unexpected breakpoints * fmt

* Fuzz coverage recording * Update cargo.toml * Update src/agent/coverage/fuzz/fuzz_targets/fuzz_target_record_coverage.rs Co-authored-by: George Pollard <porges@porg.es> * Fix fuzz --------- Co-authored-by: George Pollard <porges@porg.es>

* Reporting coverage on task start up * Moving metric up.

…com/nharper285/onefuzz into user/noharper/task-based-iterations

tevoinea

LGTM! If you want, you can update the return type of TryUpdate to Async.Task<OnefuzzResultVoid> to communicate the fallibility of the function instead of throwing but don't consider this a blocking comment.

src/ApiService/ApiService/onefuzzlib/JobResultOperations.cs

src/ApiService/ApiService/OneFuzzTypes/Model.cs

src/ApiService/ApiService/onefuzzlib/JobResultOperations.cs

…com/nharper285/onefuzz into user/noharper/task-based-iterations

src/ApiService/ApiService/onefuzzlib/JobResultOperations.cs

AdamL-Microsoft and others added 30 commits August 29, 2023 19:57

Release 8.7.1 (hotfix) (microsoft#3459)

c69deed

* Remove the retention policy setting (microsoft#3452) --------- Co-authored-by: Cheick Keita <chkeita@microsoft.com>

Revert "Release 8.7.1 (hotfix) (microsoft#3459)" (microsoft#3468)

c8986aa

This reverts commit c69deed.

Redo 8.7.1 (microsoft#3469)

7b40402

* Redo-8.7.1-hotfix --------- Co-authored-by: Cheick Keita <chkeita@microsoft.com>

Support custom ado fields that mark work items as duplicate (microsof…

d999603

…t#3467) * Add field to ado config for checking duplicate work items * Make duplicate fields nullable and add it to python models * Update broken tests * Update docs to include new ado_duplicate_fields property

Update readme with archive message (microsoft#3408)

b2435b1

Co-authored-by: Adam <103067949+AdamL-Microsoft@users.noreply.github.com>

Remove unnecessary method argument (microsoft#3473)

f141050

Updating requirements.txt to accept >= onefuzztypes. (microsoft#3477)

b2e6a07

* Updating requirements.txt to accept >= onefuzztypes. * Trying to loosen restriction.

Bump azure_* crates (microsoft#3478)

74475cc

Release 8.8.0 (microsoft#3466)

64699ed

* Release 8.8.0

Fix sed checks for CLI versioning (microsoft#3486)

830b479

* Fix sed checks for CLI versioning * Fix. * Fix. * Changing build_cli * Trying greater than * Tring once more. * Trying major minor * trying to replace major minor * Using major minor

Improve handling of unexpected breakpoints (microsoft#3493)

d009476

* Improve handling of unexpected breakpoints * fmt

Update azure_* crates (microsoft#3503)

18f2b4a

Fuzz coverage recording (microsoft#3322)

9ede0de

* Fuzz coverage recording * Update cargo.toml * Update src/agent/coverage/fuzz/fuzz_targets/fuzz_target_record_coverage.rs Co-authored-by: George Pollard <porges@porg.es> * Fix fuzz --------- Co-authored-by: George Pollard <porges@porg.es>

Reporting coverage on task start up (microsoft#3502)

cde6a19

* Reporting coverage on task start up * Moving metric up.

Remove feature flag from heartbeat metrics. (microsoft#3505)

1fb1563

Update archive notice. (microsoft#3507)

c7a9827

Add onefuzz service version to job created events (microsoft#3504)

58da7b4

nharper285 added 4 commits October 17, 2023 21:55

Comparing timestamps.

09bc214

Merge branch 'main' into user/noharper/task-based-iterations

00fdd2d

Setting propery.

85b88a1

Merge branch 'user/noharper/task-based-iterations' of https://github.…

813e702

…com/nharper285/onefuzz into user/noharper/task-based-iterations

tevoinea approved these changes Oct 18, 2023

View reviewed changes

stishkin reviewed Oct 18, 2023

View reviewed changes

src/ApiService/ApiService/onefuzzlib/JobResultOperations.cs Show resolved Hide resolved

stishkin reviewed Oct 18, 2023

View reviewed changes

src/ApiService/ApiService/OneFuzzTypes/Model.cs Show resolved Hide resolved

stishkin reviewed Oct 18, 2023

View reviewed changes

src/ApiService/ApiService/onefuzzlib/JobResultOperations.cs Show resolved Hide resolved

Merge branch 'main' into user/noharper/task-based-iterations

b7bccfa

stishkin approved these changes Oct 18, 2023

View reviewed changes

Merge branch 'main' into user/noharper/task-based-iterations

c45bcd4

Porges reviewed Oct 19, 2023

View reviewed changes

src/ApiService/ApiService/onefuzzlib/JobResultOperations.cs Outdated Show resolved Hide resolved

nharper285 added 7 commits October 24, 2023 10:50

Merge branch 'main' into user/noharper/task-based-iterations

5983596

Adding check for CreatedAt.

ba2be61

Merge branch 'user/noharper/task-based-iterations' of https://github.…

4e9dfb8

…com/nharper285/onefuzz into user/noharper/task-based-iterations

Making created_at optional.

52c1156

Trying again.

42d30f7

Merge branch 'main' into user/noharper/task-based-iterations

ad83ea5

Merge branch 'main' into user/noharper/task-based-iterations

7109c62

Porges reviewed Oct 25, 2023

View reviewed changes

src/ApiService/ApiService/onefuzzlib/JobResultOperations.cs Outdated Show resolved Hide resolved

Porges reviewed Oct 25, 2023

View reviewed changes

src/ApiService/ApiService/onefuzzlib/JobResultOperations.cs Outdated Show resolved Hide resolved

Porges reviewed Oct 25, 2023

View reviewed changes

src/ApiService/ApiService/onefuzzlib/JobResultOperations.cs Show resolved Hide resolved

nharper285 added 7 commits October 25, 2023 19:55

Remove.

50ef2f8

Add log statement.

c9859e8

Remove function handle.:

53c5c32

Updating

9b5c40f

Concat names

a3537ba

Set version.

4d5c87d

Merge branch 'main' into user/noharper/task-based-iterations

99818f9

nharper285 merged commit 909b130 into microsoft:main Oct 26, 2023
24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JobResult Table Re-design - Improved task- and machine- granularity #3539

JobResult Table Re-design - Improved task- and machine- granularity #3539

nharper285 commented Oct 2, 2023

tevoinea left a comment

JobResult Table Re-design - Improved task- and machine- granularity #3539

JobResult Table Re-design - Improved task- and machine- granularity #3539

Conversation

nharper285 commented Oct 2, 2023

Summary of the Pull Request

PR Checklist

Info on Pull Request

Validation Steps Performed

tevoinea left a comment

Choose a reason for hiding this comment