Enhancement: Trim the indexer images and use the sandbox instead of custom dockers #367

tzaffi · 2022-08-03T23:32:03Z

Utilize New Sandbox-based testing SDK and Obsolete Steps

Refactoring make docker-test using pure-shell script test-harness.sh that uses the sandbox instead of the old docker scripts
- NOTE: an attempt has been made to make this easy to re-use in other SDK's with slight modification
Removing steps that belong to features being removed in refactored SDK testing

Defining Issue

algorand/algorand-sdk-testing#221

TODO

following a conversation with @michaeldiamant - .test-env should have the bare essentials needed to control this SDK's test setup and run. It should not overwrite SDK Testing's .env. This is because the SDK Testing repo has the authority of what branches of go-algorand, indexer, etc. are relevant for a particular feature being tested.
create an issue in the sdk-testing repo to improve CI by standing up a test-harness with a simple call to up.shas a github-actions based integrations test. CREATED: https://github.com/algorand/go-algorand-internal/issues/2235
Has trim-indexer branch of algorand-sdk-testing been merged?
- Repoint the dependency here to algorand-sdk-testing/master
- Now we can merge

At a High Level

make docker-test triggers the following:

A. make harness which calls ./test-harness.sh (replacing run_integration.sh):
1. clone the SDK Testing Repo into ./test-harness/* (renamed from /test)
2. copy the env vars in .test-env to ./test-harness/.env (this is brand new)
3. copy the cloned features into ./tests/features/* (same as previous)
4. run the cloned ./test-harness/up.sh which now does the following:
- a. use .test-env to interpolate a sandbox config template into a temporary file ./test-harness/_config.harness
  - NOTE: just as previously, there is logic to distinguish between TYPE channel v. source
  - NOTE: there is also logic to handle the quirkiness of sed on mac os versus linux
- b. clone the sandbox repo into ./test-harness/.sandbox/*
- c. copy .test-env (via its copy /test-harness/.env) into ./test-harness/.sandbox/.env which serves as a Docker-compatible env file
- d. move the populated config template into ./test-harness/.sandbox/config.harness
- e. stand up the sandbox with ./sandbox up -v harness
B. make docker-pysdk-build: builds this repo's DockerFile into the image py-sdk-testing
C. make docker-pysdk-run: executes the py-sdk-testing which runs all cucumber tests

Summary of Changes

.test-env is introduced to configure the test build and the cloned sandbox. During the build it

Step Removals

Please refer to the step removal guide in the companion SDK Testing PR.

Additional Steps being modified due to indexer v1 removal

[@rekey_v1 and @send tags] - then("the transaction should go through")
- DO NOT assert "type" in context.acl.transaction_by_id(context.txn.get_txid()) because this relies on indexer v1
[@applications.verified tag] - step("I wait for the transaction to be confirmed.")
- DO NOT assert "type" in context.acl.transaction_by_id(context.app_txid)

Additional Steps being removed due to Unused Steps Analysis

@when("we make a SearchForApplications call with {application_id} and {round}")
@when("we make a SearchForApplications call with {application_id} and {round}")
@then("Every transaction works with asset-id {assetid}")
@then('the parsed response should equal "{jsonfile}".')
@then('we expect the path used to be "{path}"')
@when("I create the payment transaction")
@when('I read a transaction "{txn}" from file "{num}"')
@when("I write the transaction to file")
@then("the transaction should still be the same")
@then("I do my part")
@given('key registration transaction parameters {fee} {fv} {lv} "{gh}" "{votekey}" "{selkey}" {votefst} {votelst} {votekd} "{gen}" "{note}"')
@when("I create the key registration transaction")

Statistics

Build time average case:

latest 5.5 minutes (see the next section)
former 7.5 minutes

Running log of build times

Py 3.10 @ 3,000 rounds: 5m 37s
Py 3.10 @ 30,000 rounds FIRST TIME: 6m 18s
Py 3.10 @ 30,000 rounds SECOND TIME: 5m 41s
"" "" THIRD TIME: 6m 8s
Py 3.10 @ 30K rounds + w/o indexer v1 steps: 5m 34s
"" "" "": 5m 35s
"" "" "" (but after sandbox bugfix): 5m 49s
"" "" "" 7m 42s :-( ...not sure if that's a blip or a new trend
"" "" "" 7m 37s ... seems to be a new unfortunate trend

Testing

No functional changes are introduced in this PR. Therefore, the currently running tests are sufficient but should be tested locally in addition to CI.

test-harness.sh

Makefile

…to trim-indexer

Makefile

.test-env

test-harness.sh

michaeldiamant

@tzaffi Excited to bring in these changes - appreciate the effort! ☕

Co-authored-by: Michael Diamant <michaeldiamant@users.noreply.github.com>

…to trim-indexer

tzaffi

oopsies

Makefile

tzaffi · 2022-08-21T23:53:24Z

Makefile

-	behave --tags=$(INTEGRATIONS) tests -f progress2
+	behave --tags=$(INTEGRATION_TAGS) tests -f progress2 --no-capture
+
+display-all-python-steps:


generates text used by the Unused Steps Analysis script

tzaffi · 2022-08-22T00:17:56Z

Makefile

@@ -1,11 +1,24 @@
-UNITS = "@unit.abijson or @unit.abijson.byname or @unit.algod or @unit.algod.ledger_refactoring or @unit.applications or @unit.atc_method_args or @unit.atomic_transaction_composer or @unit.dryrun or @unit.dryrun.trace.application or @unit.feetest or @unit.indexer or @unit.indexer.ledger_refactoring or @unit.indexer.logs or @unit.offline or @unit.rekey or @unit.transactions.keyreg or @unit.responses or @unit.responses.231 or @unit.tealsign or @unit.transactions or @unit.transactions.payment or @unit.responses.unlimited_assets or @unit.sourcemap"
+UNIT_TAGS :=  "$(subst :, or ,$(shell awk '{print $2}' tests/unit.tags | paste -s -d: -))"
+INTEGRATION_TAGS := "$(subst :, or ,$(shell awk '{print $2}' tests/integration.tags | paste -s -d: -))"


cucumber tags now reside in:

tests/unit.tags

tests/integration.tags

.test-env

michaeldiamant

Approval assumes #367 (comment) is addressed prior to merge.

Co-authored-by: Michael Diamant <michaeldiamant@users.noreply.github.com>

* add check to desc so we dont output null if undefined (#368) * Bumped version to v1.16.1 * Enhancement: Trim the indexer images and use the sandbox instead of custom dockers (#367) Co-authored-by: algochoi <86622919+algochoi@users.noreply.github.com> Co-authored-by: Michael Diamant <michaeldiamant@users.noreply.github.com> * Bug-fix: Pass verbosity through to testing harness (#373) * Enhancement: Add State Proof support (#370) * add stateproof support * Enhancement: Deprecating use of langspec (#371) * Bumped version to v1.17.0 * Update README.md Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> * Mergeback: Release v1.17.0b1 (#378) * Bumped version * Update README.md * bumped version to v1.17.0 Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Barbara Poon <barbara.poon@algorand.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: algochoi <86622919+algochoi@users.noreply.github.com> Co-authored-by: Michael Diamant <michaeldiamant@users.noreply.github.com> Co-authored-by: shiqizng <80276844+shiqizng@users.noreply.github.com> Co-authored-by: Jack Smith <jack.smith@algorand.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: Lucky Baar <lucky.baar@algorand.com>

time printout

2af8a71

tzaffi marked this pull request as draft August 3, 2022 23:32

Zeph Grunschlag added 5 commits August 3, 2022 23:53

remove the integration indexers

e47317e

"no colors anymore I want them to turn..."

acf8746

bootstrap till the point of running tests

4fc4137

seems to mostly pass now that got the indexer v1 buck in the build

96e121d

shell scripts are so fragile

0a4ca51

tzaffi changed the title ~~WIP: time printout~~ WIP: Trim the indexer images and use the sandbox instead of custom dockedrs Aug 6, 2022

tzaffi changed the title ~~WIP: Trim the indexer images and use the sandbox instead of custom dockedrs~~ WIP: Trim the indexer images and use the sandbox instead of custom dockers Aug 6, 2022

tzaffi added the Enhancement label Aug 6, 2022

tzaffi changed the title ~~WIP: Trim the indexer images and use the sandbox instead of custom dockers~~ Enhancement: WIP - Trim the indexer images and use the sandbox instead of custom dockers Aug 6, 2022

Zeph Grunschlag added 6 commits August 6, 2022 00:49

simplify and refactor into Makefile

9e09c62

bug in Makefile

800e191

bug in Makefile

cd52aba

docker ps in makefile

558a042

try docker ps gin

d04e34e

expose indxer v1 and sandbox configuration

3679626

tzaffi commented Aug 7, 2022

View reviewed changes

test-harness.sh Outdated Show resolved Hide resolved

Update test-harness.sh

32e7317

tzaffi commented Aug 7, 2022

View reviewed changes

Makefile Outdated Show resolved Hide resolved

tzaffi and others added 4 commits August 7, 2022 15:45

Update Makefile

1f06edd

no cat or any other animals required

d35731f

Merge branch 'trim-indexer' of github.com:algorand/py-algorand-sdk in…

195b28f

…to trim-indexer

merged

a6e3cbc

tzaffi commented Aug 7, 2022

View reviewed changes

Makefile Outdated Show resolved Hide resolved

tzaffi and others added 4 commits August 7, 2022 15:51

revert

06119ef

revert

40c3352

python --version sanity check

b3cfc66

bring down dockers BEFORE destroying their docker-compose + misc

85afa1e

tzaffi mentioned this pull request Aug 8, 2022

Pass thru more config vars algorand/sandbox#136

Merged

make test-harness.sh more compliant

0214958

tzaffi commented Aug 16, 2022

View reviewed changes

.test-env Outdated Show resolved Hide resolved

Update .test-env

97ffb86

michaeldiamant reviewed Aug 16, 2022

View reviewed changes

test-harness.sh Outdated Show resolved Hide resolved

michaeldiamant approved these changes Aug 16, 2022

View reviewed changes

algochoi mentioned this pull request Aug 16, 2022

Enhancement: Use sandbox for SDK Testing and remove Indexer v1 steps algorand/js-algorand-sdk#623

Merged

Zeph Grunschlag and others added 7 commits August 16, 2022 22:49

remove more indexer.features steps

748817e

Update test-harness.sh

a6a2590

Co-authored-by: Michael Diamant <michaeldiamant@users.noreply.github.com>

Merge branch 'trim-indexer' of github.com:algorand/py-algorand-sdk in…

aeea8e8

…to trim-indexer

don't default to deleting local features, and introduce verbose mode

e26460f

variable for destination tests directory

70dada9

separate cucumber tags into their own files

46387c1

quotes

b9593bf

tzaffi commented Aug 19, 2022

View reviewed changes

Makefile Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

tzaffi and others added 5 commits August 19, 2022 12:42

Apply suggestions from code review

ea50a0a

Merge remote-tracking branch 'origin' into trim-indexer

121927e

display command for fuzzy matching

309a803

lintings

9b8a769

Unused Steps Analysis cleanup

0ca7e91

tzaffi commented Aug 21, 2022

View reviewed changes

unused import

39d107a

tzaffi commented Aug 22, 2022

View reviewed changes

tzaffi requested a review from michaeldiamant August 22, 2022 00:20

michaeldiamant reviewed Aug 22, 2022

View reviewed changes

.test-env Outdated Show resolved Hide resolved

michaeldiamant approved these changes Aug 22, 2022

View reviewed changes

Update .test-env

11dbdc8

Co-authored-by: Michael Diamant <michaeldiamant@users.noreply.github.com>

tzaffi merged commit ca75305 into develop Aug 22, 2022

tzaffi deleted the trim-indexer branch August 22, 2022 16:15

algojack mentioned this pull request Aug 30, 2022

FOR REVIEW ONLY - py-algorand-sdk v1.17.0 #377

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhancement: Trim the indexer images and use the sandbox instead of custom dockers #367

Enhancement: Trim the indexer images and use the sandbox instead of custom dockers #367

tzaffi commented Aug 3, 2022 •

edited

Loading

michaeldiamant left a comment

tzaffi left a comment

tzaffi Aug 21, 2022

tzaffi Aug 22, 2022

michaeldiamant left a comment

Enhancement: Trim the indexer images and use the sandbox instead of custom dockers #367

Enhancement: Trim the indexer images and use the sandbox instead of custom dockers #367

Conversation

tzaffi commented Aug 3, 2022 • edited Loading

Utilize New Sandbox-based testing SDK and Obsolete Steps

Defining Issue

TODO

At a High Level

Summary of Changes

Step Removals

Additional Steps being modified due to indexer v1 removal

Additional Steps being removed due to Unused Steps Analysis

Statistics

Running log of build times

Testing

michaeldiamant left a comment

Choose a reason for hiding this comment

tzaffi left a comment

Choose a reason for hiding this comment

tzaffi Aug 21, 2022

Choose a reason for hiding this comment

tzaffi Aug 22, 2022

Choose a reason for hiding this comment

michaeldiamant left a comment

Choose a reason for hiding this comment

tzaffi commented Aug 3, 2022 •

edited

Loading