Don't send results to queue when using PAT #38

tonywok · 2023-05-11T23:23:36Z

Rwx cloud returns the type of access token used so that we can modify the behavior of the runner strategy to noop instead of sending results to the queue.

In addition, we add a new command abq login to store a config file at ~/.abq/config.toml

RWX Access Tokens are sourced in the order of flag, env var, and then config file.

End users can explicitly supply the location of the config file via ABQ_CONFIG_FILE. By supplying a value of "", you opt-out of sourcing the access token from a config file (This is used mostly by our tests to avoid sourcing an access token from the host machine).

`abq test` (w/ PAT)

`abq test` (w/ OrgAT)

`abq login`

QA Plan

1. ✅ Create test run w/ org access token, observe instruction to run locally

% target/debug/abq test -n 1 --run-id tonytest2 -- bundle exec rspec bigtest/benchmark_rspec/benchmark_spec.rb
Generic test runner for tonytest2 started on worker 0a773c41-1846-4fc2-9117-d9e5ee739787 (worker 0, runner 1)
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Finished in 19.17 seconds (files took 0.26604 seconds to load)
500 examples, 0 failures



--------------------------------------------------------------------------------
------------------------------------- ABQ --------------------------------------
--------------------------------------------------------------------------------

Finished in 20.38 seconds (18.26 seconds spent in test code)
500 tests, 0 failures


Run the following command to replay these tests locally:


        abq test --run-id tonytest2 --worker 0 --num 1


Specify your Access Token with the RWX_ACCESS_TOKEN env variable, passing --access-token, or running `abq login`.

2. ✅ Run ABQ report, observe report (`target/debug/abq report --run-id tonytest2`)

% target/debug/abq report --run-id tonytest2                                                        
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
....................


--------------------------------------------------------------------------------
------------------------------------- ABQ --------------------------------------
--------------------------------------------------------------------------------

Finished in 0.01 seconds (18.26 seconds spent in test code)
500 tests, 0 failures

3. ✅ Run locally w/ PAT (using abq login omitted)

% target/debug/abq test -n 1 --run-id tonytest2 -- bundle exec rspec bigtest/benchmark_rspec/benchmark_spec.rb
Generic test runner for tonytest2 started on worker fe09f9ad-9cb0-4ee5-90d2-438c7b8a0a58 (worker 0, runner 1)
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Finished in 19.06 seconds (files took 0.46623 seconds to load)
500 examples, 0 failures



--------------------------------------------------------------------------------
------------------------------------- ABQ --------------------------------------
--------------------------------------------------------------------------------

Finished in 19.90 seconds (18.13 seconds spent in test code)
500 tests, 0 failures

4. ✅ Run ABQ report w/ PAT, observe no retries

% target/debug/abq report --run-id tonytest2                                                                  
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
....................


--------------------------------------------------------------------------------
------------------------------------- ABQ --------------------------------------
--------------------------------------------------------------------------------

Finished in 0.01 seconds (18.26 seconds spent in test code)
500 tests, 0 failures

5. ✅ Run locally w/ Org Token (using abq login omitted)

Generic test runner for tonytest2 started on worker 5727bad7-71ee-4c00-93b7-5feb85237b55 (worker 0, runner 1)
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Finished in 19.13 seconds (files took 0.45984 seconds to load)
500 examples, 0 failures



--------------------------------------------------------------------------------
------------------------------------- ABQ --------------------------------------
--------------------------------------------------------------------------------

Finished in 20.05 seconds (18.18 seconds spent in test code)
500 tests, 0 failures


Run the following command to replay these tests locally:


        abq test --run-id tonytest2 --worker 0 --num 1


Specify your Access Token with the RWX_ACCESS_TOKEN env variable, passing --access-token, or running `abq login`.

6. ✅ Run ABQ report w/ Org Token, observe retries

% target/debug/abq report --run-id tonytest2                                                                  
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
........................................


--------------------------------------------------------------------------------
------------------------------------- ABQ --------------------------------------
--------------------------------------------------------------------------------

Finished in 0.01 seconds (36.44 seconds spent in test code)
500 tests, 0 failures, 500 retried

Retries:

    worker 0:
      500   ./bigtest/benchmark_rspec/benchmark_spec.rb

ayazhafiz

Looks great!

crates/abq_cli/src/main.rs

github-actions · 2023-05-12T18:48:02Z

✅ Bigtest for aa8cb1a (run)

Benchmarks:

RSpec: 8.45% overhead
- RSpec time: 17.88 seconds
- ABQ time: 19.39 seconds
RSpec parallel, 10 runs: max 12.08% overhead
- min 6.21% overhead
- standard deviation: 2.02%
Jest: 5.39% overhead
- Jest time: 21.164 seconds
- ABQ time: 22.305 seconds

Fuzz result sizes:

PASSED

Rwx cloud returns the type of access token used so that we can modify the behavior of the runner strategy to noop instead of sending results to the queue.

* Require that remote queues discover an access token * Indicate the usage of a flag, envvar or abq login

Avoids blocking in debug builds of the async executor.

…queue

github-actions · 2023-05-30T22:43:08Z

✅ Bigtest for d4bee46 (run)

Benchmarks:

RSpec: 7.64% overhead
- RSpec time: 17.79 seconds
- ABQ time: 19.15 seconds
RSpec parallel, 10 runs: max 15.68% overhead
- min 6.91% overhead
- standard deviation: 2.75%
Jest: 4.42% overhead
- Jest time: 20.435 seconds
- ABQ time: 21.338 seconds

Fuzz result sizes:

PASSED

github-actions · 2023-05-30T23:50:59Z

✅ Bigtest for 7d02df2 (run)

Benchmarks:

RSpec: 7.93% overhead
- RSpec time: 17.78 seconds
- ABQ time: 19.19 seconds
RSpec parallel, 10 runs: max 15.52% overhead
- min 6.86% overhead
- standard deviation: 2.40%
Jest: 4.24% overhead
- Jest time: 20.405 seconds
- ABQ time: 21.27 seconds

Fuzz result sizes:

PASSED

ayazhafiz

🔥

crates/abq_cli/src/abq_config.rs

crates/abq_cli/src/main.rs

crates/abq_cli/src/workers.rs

crates/abq_cli/tests/cli.rs

crates/abq_cli/tests/snapshots/cli__retries_displays_retry_banners.snap

crates/abq_workers/src/results_handler.rs

ayazhafiz

Sorry, I meant to comment without explicit approval - my bad!

* Passing an ABQ_CONFIG_FILE during abq login is now an error. * Improve worker repro message * Misc rustisms

This reverts commit b640072.

github-actions · 2023-05-31T23:29:43Z

❌ Bigtest for b640072 (run)

Benchmarks:

RSpec: % overhead
- RSpec time: seconds
- ABQ time: seconds
RSpec parallel, 10 runs: max % overhead
- min % overhead
- standard deviation: %
Jest: % overhead
- Jest time: seconds
- ABQ time: seconds

Fuzz result sizes:

PASSED

github-actions · 2023-05-31T23:48:59Z

✅ Bigtest for 7aeb351 (run)

Benchmarks:

RSpec: 11.66% overhead
- RSpec time: 17.84 seconds
- ABQ time: 19.92 seconds
RSpec parallel, 10 runs: max 16.31% overhead
- min 6.28% overhead
- standard deviation: 3.48%
Jest: 6.31% overhead
- Jest time: 21.379 seconds
- ABQ time: 22.727 seconds

Fuzz result sizes:

PASSED

ayazhafiz

Incredible work here Tony, congrats!! And thanks for this great feature!

TAGraves

Great work on this @tonywok and @ayazhafiz!

This reverts commit 89840e6.

ayazhafiz reviewed May 11, 2023

View reviewed changes

crates/abq_cli/src/main.rs Outdated Show resolved Hide resolved

tonywok force-pushed the ts/local-repro-wo-updating-queue branch 3 times, most recently from 3dd8952 to aa8cb1a Compare May 12, 2023 18:25

tonywok force-pushed the ts/local-repro-wo-updating-queue branch from 72d877d to 47ba6f5 Compare May 22, 2023 13:55

tonywok force-pushed the ts/local-repro-wo-updating-queue branch 3 times, most recently from 24bc25a to 17115ab Compare May 30, 2023 16:09

tonywok and others added 16 commits May 30, 2023 13:19

Don't send results to queue when using PAT

1acc3f5

Rwx cloud returns the type of access token used so that we can modify the behavior of the runner strategy to noop instead of sending results to the queue.

Add enum for AccessTokenKind

0b2f38a

lint

572a710

very rough sketch of login command and usage

fa4b145

extract abq_config module

208627f

print run command from test when executing in write normal mode

fdb9f8a

clean up messaging

cd421f1

lint

c5d7b5c

Feedback on messaging

fb8eae3

Extract a QueueLocation concept

9bd8453

* Require that remote queues discover an access token * Indicate the usage of a flag, envvar or abq login

Standing up integration test

3ad2c49

Refactor all hosted functions to be async

705782c

Avoids blocking in debug builds of the async executor.

lint

09911a3

moar lint

f8bb21f

moar lint

7e6411a

Cover errors where we've been provided a run id against an ephemeral …

17b3c40

…queue

tonywok force-pushed the ts/local-repro-wo-updating-queue branch from a69b903 to 17b3c40 Compare May 30, 2023 17:21

tonywok added 4 commits May 30, 2023 13:26

hoisting helpers to the top

e8969f1

lint

c3b0a82

add spec for abq config file read/write

3b3151a

lint

36bf55a

tonywok self-assigned this May 30, 2023

Add test verifying file missing results in no config

7d02df2

tonywok marked this pull request as ready for review May 31, 2023 00:38

ayazhafiz previously approved these changes May 31, 2023

View reviewed changes

ayazhafiz reviewed May 31, 2023

View reviewed changes

ayazhafiz self-requested a review May 31, 2023 03:21

PR Feedback

bdc4623

* Passing an ABQ_CONFIG_FILE during abq login is now an error. * Improve worker repro message * Misc rustisms

tonywok dismissed ayazhafiz’s stale review via bdc4623 May 31, 2023 16:54

tonywok force-pushed the ts/local-repro-wo-updating-queue branch from 05958be to efbdfc8 Compare May 31, 2023 19:33

Clarify runner strategy implementation and fix output spacing

f05c809

tonywok force-pushed the ts/local-repro-wo-updating-queue branch 3 times, most recently from 674ca59 to a2b8060 Compare May 31, 2023 21:58

Fix invalid test command format

018caf2

tonywok force-pushed the ts/local-repro-wo-updating-queue branch from a2b8060 to 018caf2 Compare May 31, 2023 22:06

ayazhafiz added 5 commits May 31, 2023 17:34

bump CI

7baa484

Test nocapture

a9f90b3

Ignore a test

b640072

Revert "Ignore a test"

c597675

This reverts commit b640072.

Test as normal

7aeb351

ayazhafiz approved these changes May 31, 2023

View reviewed changes

TAGraves approved these changes May 31, 2023

View reviewed changes

TAGraves merged commit 89840e6 into main May 31, 2023
17 checks passed

TAGraves deleted the ts/local-repro-wo-updating-queue branch May 31, 2023 23:53

ayazhafiz mentioned this pull request May 31, 2023

Handle launching tests on a remote queue with a personal access token #46

Closed

ayazhafiz added a commit that referenced this pull request Jun 1, 2023

Revert "Don't send results to queue when using PAT (#38)"

31a80d3

This reverts commit 89840e6.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't send results to queue when using PAT #38

Don't send results to queue when using PAT #38

tonywok commented May 11, 2023 •

edited

ayazhafiz left a comment

github-actions bot commented May 12, 2023

github-actions bot commented May 30, 2023

github-actions bot commented May 30, 2023

ayazhafiz left a comment

ayazhafiz left a comment

github-actions bot commented May 31, 2023

github-actions bot commented May 31, 2023

ayazhafiz left a comment

TAGraves left a comment

Don't send results to queue when using PAT #38

Don't send results to queue when using PAT #38

Conversation

tonywok commented May 11, 2023 • edited

abq test (w/ PAT)

abq test (w/ OrgAT)

abq login

QA Plan

ayazhafiz left a comment

Choose a reason for hiding this comment

github-actions bot commented May 12, 2023

github-actions bot commented May 30, 2023

github-actions bot commented May 30, 2023

ayazhafiz left a comment

Choose a reason for hiding this comment

ayazhafiz left a comment

Choose a reason for hiding this comment

github-actions bot commented May 31, 2023

github-actions bot commented May 31, 2023

ayazhafiz left a comment

Choose a reason for hiding this comment

TAGraves left a comment

Choose a reason for hiding this comment

tonywok commented May 11, 2023 •

edited

`abq test` (w/ PAT)

`abq test` (w/ OrgAT)

`abq login`