Fix part of #2691: Migrate compute affected tests script to Kotlin #3374

BenHenning · 2021-06-26T02:44:00Z

Fix part of #2691

Explanation

This PR introduces a new version of the compute_affected_tests script that exists in Kotlin instead of Bash.

The main reason for doing this is the new script includes code to fix situations where Bazel outputs malformed query results (e.g. when two targets are on the same line). It's not yet clear whether these issues are caused by Bazel or Bash, but they only happen in CI (i.e. can't be reproduced locally) on occasional PRs (such as #3340). This version of the script builds in resilience to such failures.

There are a number of other benefits that we see moving to Kotlin, as well:

It's easier to test the script
The script is normalizing on an existing language well understood in the codebase (Kotlin) vs. using an alternative language (such as Python)
The script is easier to understand since advanced Bash syntax is hard to read and understand vs. Kotlin which is a much more self-documenting language
The script establishes the foundation for introducing more Kotlin-based scripts in the future

Beyond that, three other things to note:

I noticed that the per-file query loop in the original Bash script can be simplified to a single command which seems to be more performant. Beyond that, the script overall seems much faster through Bazel vs. Bash (though CI will be the true test, especially since the binary also needs to be build via Bazel itself).
The linters have been updated to include the scripts directory as actual code
I adjusted the output of the ktlint command to include a copy-and-pastable line for fixing errors based on the version of ktlint used to report the errors
The command runs in a different directory than it did with Bash hence the updated usage arguments. This is actually safer from a sandboxing perspective as it reduces the chance of the script having accidental side effects in the repository.
The original command may have had a bug in it when an empty set is passed for BUILD files (such as if the script is run with no files changed). This has been addressed in the new script.
The original script would sometimes hang on CI for unknown reasons. While this could still happen, additional debug lines have been added to the script to help with this & other issues as they crop up. Further, the new script has a timeout if subcommands take longer than 60 seconds, so CI should never indefinitely hang due to the script now.
The new script is actually testable, and the new tests are creating a real Bazel & Git repository locally in order to simulate different situations.
The new script is slower since it's no longer in Bash, but it seems fast enough for our purpose (it actually seems to be comparable, but this PR is simpler from a dependency graph perspective than some PRs we expect this to run in).

Long-term, we'll likely want to move all of our scripts over to Kotlin to reduce the total number of languages needed for the repository by 1.

Checklist

The PR title starts with "Fix #bugnum: ", followed by a short, clear summary of the changes. (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".)
The PR explanation includes the words "Fixes #bugnum: ..." (or "Fixes part of #bugnum" if the PR only partially fixes an issue).
The PR follows the style guide.
The PR does not contain any unnecessary auto-generated code from Android Studio.
The PR is made from a branch that's not called "develop".
The PR is made from a branch that is up-to-date with "develop".
The PR's branch is based on "develop" and not on any other branch.
The PR is assigned to an appropriate reviewer in both the Assignees and the Reviewers sections.

No tests for the script yet.

BenHenning · 2021-06-26T02:45:34Z

@Sparsh1212 are you still hitting the problem we discussed in #3340? It might be nice to do a trial run on your PR, but that might be tricky since the code needs to be copied into the branch and uploaded to GitHub.

Keeping this as a draft until I write tests, though keep in mind that tests are particularly hard for this script since it's highly dependent on a real Bazel environment.

Sparsh1212 · 2021-06-26T06:27:32Z

@BenHenning I copied the code from regex-checks branch to this branch and initiated a draft PR to see the CI checks are working properly or not.
PR: #3375

Sparsh1212 · 2021-06-26T06:33:57Z

The parsing of the added kt_jvm_test: test_pattern_validation_check is now correctly taking place. Also, the tests are passing as well.

BenHenning · 2021-06-26T07:03:07Z

Ah excellent--thanks @Sparsh1212. In that case, this just needs tests & can then go into review.

Sparsh1212 · 2021-06-26T07:04:39Z

Ah excellent--thanks @Sparsh1212. In that case, this just needs tests & can then go into review.

Thanks! for the fix Ben!

Fix some typos & realized bug in the script.

BenHenning · 2021-06-30T03:18:20Z

Added tests for the core script. Still need to refactor, added more documentation, and add tests for the test & script utilities.

BenHenning · 2021-06-30T03:19:22Z

Also interesting: I changed the merge-base command to use develop instead of origin/develop since the test repository doesn't have an origin, but that doesn't work in the CI environment. The CI script probably needs to check out the develop branch so that it exists locally.

See comment in PR for context.

BenHenning · 2021-06-30T03:24:53Z

Unfortunately the extra checkout action didn't work. We may need to pass the reference object for merge-base in as a parameter unfortunately.

This splits up the test & script into multiple utility packages (for common & testing utilities), reorganizes the test to be in a proper javatests/ & java/ arrangement, and adds tests for checking the error conditions of the test.

This also refactors that function to be part of a class so that its operating conditions are configurable (namely the process timeout).

Also, fix a few small issues in the utility.

Also, fix some changed behaviors that broke the affected tests script.

BenHenning · 2021-07-03T01:08:32Z

I was able to isolate the problem. I've added context to bazelbuild/bazel#906. I might need to file a new issue since that one is old, but I'm really interested in seeing whether the isolated repository will make it easier to investigate the issue.

In the meantime, I'll disable the test with a TODO to unblock this & downstream work.

See #2691 for context.

BenHenning · 2021-07-03T01:13:28Z

Now that the test is ignored, I expect all remaining tests to pass. Will reassign once I confirm the failing suite in question is now passing on CI.

BenHenning · 2021-07-03T01:17:57Z

BazelClientTest is now passing: https://github.com/oppia/oppia-android/pull/3374/checks?check_run_id=2976899202. Assigning this back over since the PR should have a fully green CI now.

BenHenning · 2021-07-03T01:18:14Z

@Sparsh1212 / @anandwana001 PTAL at my follow-up comments & mark as resolved if you're happy with the changes made.

vinitamurthi

LGTM!

Sparsh1212 · 2021-07-03T05:05:40Z

LGTM!

oppiabot · 2021-07-03T05:10:03Z

Unassigning @Sparsh1212 since they have already approved the PR.

anandwana001

LGTM

BenHenning · 2021-07-03T22:19:44Z

Thanks all! I might follow up with a workaround to the issue with the query hanging (I have an idea that might mitigate it).

* Fix #3290: Add support for generic regex pattern matching * Apply review suggestions for proto location * Delete main directory * Implment script logic and tests * CI setup part 1 * syntax fix in yaml file * import nits in dummy file * Use regex patterns from script in script test * Make PR suggestions into effect * Make PR suggestions into effect * Make PR suggestions into effect * Improve naming of script_assets variables * Make PR suggestions into effect * Make PR suggestions * Make PR suggestions * Revamp testing approach * nit fixes * Make nit suggestions * Apply review suggestions on PR * Refactor PR as per feedback recieved * Implement review suggestions * nits * Nit fixes * nit fix * Review suggestions part 1 * Implement review suggestions part 2 * nit fixes * update static_checks * add test to repository file * nit fix * nit fix * nit fix * Implement review suggestions * bazel files nit fix * Introduce a library for the regex assets * Make directory structure same as that of #3374 * diable ktlint max-line-length * disable ktlint-max-line * disable ktlint max-length * nit fix * nit fix * Apply review suggestions * ktlint fix * nit fix * Remove script constants * nit fix * add todo * nit fixes * add testOnly

* Fix #3290: Add support for generic regex pattern matching * Apply review suggestions for proto location * Delete main directory * Implment script logic and tests * CI setup part 1 * syntax fix in yaml file * import nits in dummy file * Use regex patterns from script in script test * Make PR suggestions into effect * Make PR suggestions into effect * Make PR suggestions into effect * Improve naming of script_assets variables * Make PR suggestions into effect * Make PR suggestions * Make PR suggestions * Revamp testing approach * Fix #3291: Add check for XML syntax correctness * nit fixes * Introduce XML syntax check in static checks workflow * Make nit suggestions * Make nit suggestions * Add if: always() * Apply review suggestions on PR * Make review suggestions from regex pattern checks * Refactor PR as per feedback recieved * Implement feedback suggestions * Implement review suggestions * Implement review suggestions * nits * Nit fixes * Nit fixes * nit fix * nit fix * Review suggestions part 1 * Implement review suggestions part 2 * nit fixes * update static_checks * add test to repository file * nit fix * nit fix * nit fix * Do review suggestions * Implement review suggestions * Implement review suggestions * nit fix * bazel files nit fix * Introduce a library for the regex assets * Make directory structure same as that of #3374 * Make directory structure similar to #3374 * diable ktlint max-line-length * disable ktlint-max-line * disable ktlint max-length * remove testonly attribute from tests * nit fix * nit fix * Apply review suggestions * ktlint fix * nit fix * Apply nit fixes * Remove script constants * nit fix * add todo * nit fixes * add test * add testOnly * nit fix

* Fix #3290: Add support for generic regex pattern matching * Apply review suggestions for proto location * Delete main directory * Implment script logic and tests * CI setup part 1 * syntax fix in yaml file * import nits in dummy file * Use regex patterns from script in script test * Make PR suggestions into effect * Make PR suggestions into effect * Make PR suggestions into effect * Improve naming of script_assets variables * Make PR suggestions into effect * Make PR suggestions * Make PR suggestions * Revamp testing approach * Fix #3291: Add check for XML syntax correctness * nit fixes * Introduce XML syntax check in static checks workflow * Fix #3292: Add check for test files presence for prod files * Make nit suggestions * Make nit suggestions * Make nit suggestions * Add ScriptConstants file to exemptions test list * Add if: always() * Add if: always() * Apply review suggestions on PR * Make review suggestions from regex pattern checks * Implement review suggestions based on #3340 * Refactor PR as per feedback recieved * Implement feedback suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * nits * Nit fixes * Nit fixes * Nit fixes * nit fix * nit fix * Review suggestions part 1 * Implement review suggestions part 2 * nit fixes * update static_checks * add test to repository file * nit fix * nit fix * nit fix * Do review suggestions * Implement review suggestions * Implement review suggestions * nit fix * Implement review suggestions * Add Ben as a codeowner for the ScriptExemptions file * bazel files nit fix * Introduce a library for the regex assets * Make directory structure same as that of #3374 * Make directory structure similar to #3374 * add new files to test file script exemptions and nit fixes * diable ktlint max-line-length * disable ktlint-max-line * disable ktlint max-length * remove testonly attribute from tests * add new files to script exemptions * nit fix * nit fix * Apply review suggestions * ktlint fix * nit fix * Apply nit fixes * Apply nit fixes * Remove script constants * nit fix * add todo * add todo * nit fixes * add test * add testOnly * nit fixes * nit fix * nit fixes * nit fixes * nit fixes * nit fixes * fix textproto file * add exemptions to textproto * sort failures lexicographically * add new files to exemption list

… and license links (#3434) * Generate maven_install.json * Add fail_if_repin_required attribute * Update rules_jvm_external version in versions.bzl and use fail_if_repin_required attribute in maven_install() * Fix bazel lint error. * Update code ownership for maven_install.json * Fix #3290: Add support for generic regex pattern matching * Apply review suggestions for proto location * Delete main directory * Implment script logic and tests * CI setup part 1 * syntax fix in yaml file * import nits in dummy file * Use regex patterns from script in script test * Make PR suggestions into effect * Make PR suggestions into effect * Make PR suggestions into effect * Improve naming of script_assets variables * Make PR suggestions into effect * Create maven_dependencies_list.kts and complete the parsing of the maven_install.json * Make PR suggestions * Make PR suggestions * Improve naming in script and add KDocs for data classes. * Add EOF for all classes. * Change data structure to store backup dependencies. * Revamp testing approach * nit fixes * Add maven re-pin command. * Add BUILD.bazel file and Test file for script.creatre-script-to-compile-list-of-maven-dependencies * Fix test build error. * Improve code. * Make nit suggestions * Try changing location of maven_install.json * Convert .kts to .kt * Save work. * Rename data classes. * Remove kscript. * Fix lints. * Apply review suggestions on PR * Refactor PR as per feedback recieved * Implement review suggestions * Set different output_base for bazel query command * nits * Nit fixes * nit fix * Change maven_install_json attribute * Move maven_install.json to third_party * Remove maven_install code ownership from CODEOWNERS * Save changes. * remove class * Fix errors * Review suggestions part 1 * Implement review suggestions part 2 * Fix broken script * nit fixes * update static_checks * add test to repository file * nit fix * nit fix * Validate all Links. * nit fix * Implement review suggestions * bazel files nit fix * Introduce a library for the regex assets * Change structure of the script. * Make directory structure same as that of #3374 * diable ktlint max-line-length * disable ktlint-max-line * disable ktlint max-length * save changes. * commit * nit fix * Add bazel pipeline to app module * save. * save. * save. * save. * save. * save. * nit fix * fix broken script * Fix bug of printing enum values in output. * Fix bug of printing enum values in output. * Apply review suggestions * ktlint fix * nit fix * Remove script constants * nit fix * Make script to call out the dependencies and licenses for which manual work needs to be completed. * add todo * Complete first script * Add first script to scripts package * Remove unwanted files * Remove extra files and fix formatting * Fix lints. * Remove unnecessary lines of code * Add comments for proto * Add test file. * Fix lints. * Remove unwanted dependencies. * Add suggested changes. * nit fixes * Fix lint. * add testOnly * Fix nits * Add suggested changes. * Add suggested changes * Update textproto * Update maven_dependencies.textproto * Always force https over http * Always force https over http * Correct origin type * Use common/BazelClient * Fix lints * Save progress * create test_maven_install.json * Restructure script * Break script in two parts * Add DependencyListsProvider * Add DependencyListsProvider * correct query command * Correct structure of the script * Correct structure of the script * Fix not suggestions * Use * Add test file for NetworkAndBazelUtilsImpl.kt * Add KDocs * Add KDocs * try using mockito * Add test case in BazelClientTest.kt * Add mockito-kotlin * Fix nit * Fix nit * try fixing tests * Restructure maven_dependencies.proto * Fix lint * add --max_idle_secs * Fix nits * Fix nit * Place KDoc at correct position * Fix BazelClientTest.kt * Add xml parser * Fix pb error * Revert data module changes * Add tests * Add first test that passes * Add all test cases. * Fix nit * Add suggested changes * Add suggested changes * Add suggested changes * Nit changes * Fix nits * Fix BazelClient test case * Add test cases for TestBazelWorkspace * Add suggested changes * Add suggested changes * Fix one test. * Fix all test cases * Add TODO for issue #3486 * Add suggestions. * Fix urls. * Fix isFalse * Add all suggested changes. * Fix nits. Co-authored-by: Sparsh1212 <sparshagrawal1212@gmail.com>

* Fix #3290: Add support for generic regex pattern matching * Apply review suggestions for proto location * Delete main directory * Implment script logic and tests * CI setup part 1 * syntax fix in yaml file * import nits in dummy file * Use regex patterns from script in script test * Make PR suggestions into effect * Make PR suggestions into effect * Make PR suggestions into effect * Improve naming of script_assets variables * Make PR suggestions into effect * Make PR suggestions * Make PR suggestions * Revamp testing approach * Fix #3291: Add check for XML syntax correctness * nit fixes * Introduce XML syntax check in static checks workflow * Fix #3292: Add check for test files presence for prod files * Make nit suggestions * Make nit suggestions * Make nit suggestions * Fix #3300: Add check for accessibility labels for activities * Add CI check for label presence for activities * Add ScriptConstants file to exemptions test list * Test if: always() * Revert "Test if: always()" This reverts commit b72e419 * Add if: always() * Add if: always() * Add if: always() * Apply review suggestions on PR * Make review suggestions from regex pattern checks * Implement review suggestions based on #3340 * Refactor PR as per feedback recieved * Implement feedback suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * nits * Nit fixes * Nit fixes * Nit fixes * Implement feedback suggestions as per #3340 * nit fix * nit fix * nit fix * nit fix * Review suggestions part 1 * Implement review suggestions part 2 * nit fixes * update static_checks * add test to repository file * nit fix * nit fix * nit fix * Do review suggestions * Implement review suggestions * Implement review suggestions * nit fix * Implement review suggestions * Add Ben as a codeowner for the ScriptExemptions file * bazel files nit fix * Introduce a library for the regex assets * Make directory structure same as that of #3374 * Make directory structure similar to #3374 * add new files to test file script exemptions and nit fixes * diable ktlint max-line-length * disable ktlint-max-line * disable ktlint max-length * remove testonly attribute from tests * add new files to script exemptions * nit fix * nit fix * nit fix * Apply review suggestions * ktlint fix * nit fix * Apply nit fixes * Apply nit fixes * Remove script constants * nit fix * add todo * add todo * nit fixes * add test * add testOnly * nit fixes * nit fix * nit fixes * nit fixes * nit fixes * nit fixes * fix textproto file * add exemptions to textproto * sort failures lexicographically * add new files to exemption list * nit fixes * nit fixes * add kdocs * nit fix * nit fix * Trigger CI failure exclusively * Revert CI check failure * nit fix * nit fix * nit fixes * nit fixes * nit fixes * Add redundant exemptions check * rectify static_checks * Remove duplicate proto libraries * nit fixes * add activity to exemption * nit fixes * nit fixes * nit fixes * update branch

…3499) * Fix #3290: Add support for generic regex pattern matching * Apply review suggestions for proto location * Delete main directory * Implment script logic and tests * CI setup part 1 * syntax fix in yaml file * import nits in dummy file * Use regex patterns from script in script test * Make PR suggestions into effect * Make PR suggestions into effect * Make PR suggestions into effect * Improve naming of script_assets variables * Make PR suggestions into effect * Make PR suggestions * Make PR suggestions * Revamp testing approach * Fix #3291: Add check for XML syntax correctness * nit fixes * Introduce XML syntax check in static checks workflow * Fix #3292: Add check for test files presence for prod files * Make nit suggestions * Make nit suggestions * Make nit suggestions * Fix #3300: Add check for accessibility labels for activities * Add CI check for label presence for activities * Add ScriptConstants file to exemptions test list * Test if: always() * Revert "Test if: always()" This reverts commit b72e419 * Add if: always() * Add if: always() * Add if: always() * Apply review suggestions on PR * Make review suggestions from regex pattern checks * Implement review suggestions based on #3340 * Refactor PR as per feedback recieved * Implement feedback suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * nits * Nit fixes * Nit fixes * Nit fixes * Implement feedback suggestions as per #3340 * nit fix * nit fix * nit fix * nit fix * Review suggestions part 1 * Implement review suggestions part 2 * nit fixes * update static_checks * add test to repository file * nit fix * nit fix * nit fix * Do review suggestions * Implement review suggestions * Implement review suggestions * nit fix * Implement review suggestions * Add Ben as a codeowner for the ScriptExemptions file * bazel files nit fix * Introduce a library for the regex assets * Make directory structure same as that of #3374 * Make directory structure similar to #3374 * add new files to test file script exemptions and nit fixes * diable ktlint max-line-length * disable ktlint-max-line * disable ktlint max-length * remove testonly attribute from tests * add new files to script exemptions * nit fix * nit fix * nit fix * Apply review suggestions * ktlint fix * nit fix * Apply nit fixes * Apply nit fixes * Remove script constants * nit fix * add todo * add todo * nit fixes * add test * add testOnly * nit fixes * nit fix * nit fixes * nit fixes * nit fixes * nit fixes * fix textproto file * add exemptions to textproto * sort failures lexicographically * add new files to exemption list * nit fixes * nit fixes * add kdocs * nit fix * nit fix * Trigger CI failure exclusively * Revert CI check failure * Fix #3340: Create a script to ensure KDoc presence * Add initial tests * Add more tests and add documentation * nit fix * nit fix * nit fix * nit fix * Add more files to kdoc exemptions * nit fixes * nit fixes * nit fixes * Add redundant exemptions check * rectify static_checks * Remove duplicate proto libraries * nit fixes * nit fixes and add redundant exemptions check to script * nit fixes * add activity to exemption * nit fixes * nit fixes * nit fixes * nit fixes * nit fixes * update branch * nit fixes

* Generate maven_install.json * Add fail_if_repin_required attribute * Update rules_jvm_external version in versions.bzl and use fail_if_repin_required attribute in maven_install() * Fix bazel lint error. * Update code ownership for maven_install.json * Fix #3290: Add support for generic regex pattern matching * Apply review suggestions for proto location * Delete main directory * Implment script logic and tests * CI setup part 1 * syntax fix in yaml file * import nits in dummy file * Use regex patterns from script in script test * Make PR suggestions into effect * Make PR suggestions into effect * Make PR suggestions into effect * Improve naming of script_assets variables * Make PR suggestions into effect * Create maven_dependencies_list.kts and complete the parsing of the maven_install.json * Make PR suggestions * Make PR suggestions * Improve naming in script and add KDocs for data classes. * Add EOF for all classes. * Change data structure to store backup dependencies. * Revamp testing approach * nit fixes * Add maven re-pin command. * Add BUILD.bazel file and Test file for script.creatre-script-to-compile-list-of-maven-dependencies * Fix test build error. * Improve code. * Make nit suggestions * Try changing location of maven_install.json * Convert .kts to .kt * Save work. * Rename data classes. * Remove kscript. * Fix lints. * Apply review suggestions on PR * Refactor PR as per feedback recieved * Implement review suggestions * Set different output_base for bazel query command * nits * Nit fixes * nit fix * Change maven_install_json attribute * Move maven_install.json to third_party * Remove maven_install code ownership from CODEOWNERS * Save changes. * remove class * Fix errors * Review suggestions part 1 * Implement review suggestions part 2 * Fix broken script * nit fixes * update static_checks * add test to repository file * nit fix * nit fix * Validate all Links. * nit fix * Implement review suggestions * bazel files nit fix * Introduce a library for the regex assets * Change structure of the script. * Make directory structure same as that of #3374 * diable ktlint max-line-length * disable ktlint-max-line * disable ktlint max-length * save changes. * commit * nit fix * Add bazel pipeline to app module * save. * save. * save. * save. * save. * save. * nit fix * fix broken script * Fix bug of printing enum values in output. * Fix bug of printing enum values in output. * Apply review suggestions * ktlint fix * nit fix * Remove script constants * nit fix * Make script to call out the dependencies and licenses for which manual work needs to be completed. * add todo * Complete first script * Add first script to scripts package * Remove unwanted files * Remove extra files and fix formatting * Fix lints. * Remove unnecessary lines of code * Add comments for proto * Add test file. * Fix lints. * Add second script * Remove unwanted dependencies. * Add suggested changes. * Add test file * nit fixes * Fix lint. * add testOnly * Add pipeline to print actual dependencies' names * Fix nits * Add suggested changes. * Add suggested changes * add function to write license texts. * Update text proto * Update textproto * Update maven_dependencies.textproto * Update Licenses writing function * Write license name array * Always force https over http * Always force https over http * Correct origin type * Complete script * Fix lints. * Add support for escape characters. * Fix nits * Use common/BazelClient * Fix lints * Save progress * create test_maven_install.json * Add test cases * Create function to add license * Add more test cases * Fix lints * Correct typo * Add one more test case * Restructure script * Break script in two parts * Add DependencyListsProvider * Add DependencyListsProvider * correct query command * Correct structure of the script * Correct structure of the script * Fix not suggestions * Use * Add test file for NetworkAndBazelUtilsImpl.kt * Delete unwanted files * Delete unwanted files * Add library for GenerateLicenseTexts.kt * Add suggested changes * Add data classes to exemptions.textproto * Remove commented out code * Add KDocs for helper functions in GenerateLicenseTextsTest.kt * Remove unused variables * Delete unwanted proto file * Use createCDATASection instead of manually constructing it * Add suggested changes * Add suggested changes * Correct test case * Fix nit * Add changes as per the UI * Fix script to preserve text format of licenses * Remove extra lines of code * Add KDoc * Add suggested changes * Omit version from the dependency names in xml * Add suggested changes * Add suggested changes * Correct KDoc * Correct file paths in exemptions file * Revert unwanted chang. * Refactor 'data_lib' to 'data' * Add suggested changes * Fix lints * Correct deps in BUILD.bazel * Correct deps in BUILD.bazel * Nit fix * Nit fix * Nit fix * Nit fix * Use .map Co-authored-by: Sparsh1212 <sparshagrawal1212@gmail.com>

…nd correspond to an open issue on Github (#3508) * Fix #3290: Add support for generic regex pattern matching * Apply review suggestions for proto location * Delete main directory * Implment script logic and tests * CI setup part 1 * syntax fix in yaml file * import nits in dummy file * Use regex patterns from script in script test * Make PR suggestions into effect * Make PR suggestions into effect * Make PR suggestions into effect * Improve naming of script_assets variables * Make PR suggestions into effect * Make PR suggestions * Make PR suggestions * Revamp testing approach * Fix #3291: Add check for XML syntax correctness * nit fixes * Introduce XML syntax check in static checks workflow * Fix #3292: Add check for test files presence for prod files * Make nit suggestions * Make nit suggestions * Make nit suggestions * Fix #3300: Add check for accessibility labels for activities * Add CI check for label presence for activities * Add ScriptConstants file to exemptions test list * Test if: always() * Revert "Test if: always()" This reverts commit b72e419 * Add if: always() * Add if: always() * Add if: always() * Apply review suggestions on PR * Make review suggestions from regex pattern checks * Implement review suggestions based on #3340 * Refactor PR as per feedback recieved * Implement feedback suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * nits * Nit fixes * Nit fixes * Nit fixes * Implement feedback suggestions as per #3340 * nit fix * nit fix * nit fix * nit fix * Review suggestions part 1 * Implement review suggestions part 2 * nit fixes * update static_checks * add test to repository file * nit fix * nit fix * nit fix * Do review suggestions * Implement review suggestions * Implement review suggestions * nit fix * Implement review suggestions * Add Ben as a codeowner for the ScriptExemptions file * bazel files nit fix * Introduce a library for the regex assets * Make directory structure same as that of #3374 * Make directory structure similar to #3374 * add new files to test file script exemptions and nit fixes * diable ktlint max-line-length * disable ktlint-max-line * disable ktlint max-length * remove testonly attribute from tests * add new files to script exemptions * nit fix * nit fix * nit fix * Apply review suggestions * ktlint fix * nit fix * Apply nit fixes * Apply nit fixes * Remove script constants * nit fix * add todo * add todo * nit fixes * add test * add testOnly * nit fixes * nit fix * nit fixes * nit fixes * nit fixes * nit fixes * fix textproto file * add exemptions to textproto * sort failures lexicographically * add new files to exemption list * nit fixes * nit fixes * add kdocs * nit fix * nit fix * Trigger CI failure exclusively * Revert CI check failure * Fix #3340: Create a script to ensure KDoc presence * Add initial tests * Add more tests and add documentation * nit fix * nit fix * nit fix * nit fix * Add more files to kdoc exemptions * Implementation part 1 * add exemption proto for todos * nit fixes * delete open issues * nit fixes * nit fixes * Add redundant exemptions check * rectify static_checks * Remove duplicate proto libraries * nit fixes * nit fixes and add redundant exemptions check to script * nit fixes * add activity to exemption * nit fixes * nit fixes * nit fixes * nit fixes * revamption part 1 * nit fixes * update branch * Revamped approach * nit fixes * add kdocs * delete open_issues.json * Fix #3317: Update old todos * nit fixes * Remove resolved todos * Repurpose todos stage 1 * Repurpose todos stage 2 * updating todos part 3 * update more todos * add network module * update todo no. 322 * Remove unused imports * Revert "Remove unused imports" This reverts commit 65ee6e9. * Revert "update todo no. 322" This reverts commit c150cdc. * Repurpose no. 322 * nit fixes * replace issue number for backend model todo * link new filed issue * nit fixes * add exemptions * nit fixes * nit fixes * nit fixes * add remaining tests for all cases * nit fixes * nit fixes Co-authored-by: Sparsh Agrawal <sparsh@Sparshs-MacBook-Air.local>

…to the closed issue (#3629) * Fix #3291: Add check for XML syntax correctness * nit fixes * Introduce XML syntax check in static checks workflow * Fix #3292: Add check for test files presence for prod files * Make nit suggestions * Make nit suggestions * Make nit suggestions * Fix #3300: Add check for accessibility labels for activities * Add CI check for label presence for activities * Add ScriptConstants file to exemptions test list * Test if: always() * Revert "Test if: always()" This reverts commit b72e419 * Add if: always() * Add if: always() * Add if: always() * Apply review suggestions on PR * Make review suggestions from regex pattern checks * Implement review suggestions based on #3340 * Refactor PR as per feedback recieved * Implement feedback suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * nits * Nit fixes * Nit fixes * Nit fixes * Implement feedback suggestions as per #3340 * nit fix * nit fix * nit fix * nit fix * Review suggestions part 1 * Implement review suggestions part 2 * nit fixes * update static_checks * add test to repository file * nit fix * nit fix * nit fix * Do review suggestions * Implement review suggestions * Implement review suggestions * nit fix * Implement review suggestions * Add Ben as a codeowner for the ScriptExemptions file * bazel files nit fix * Introduce a library for the regex assets * Make directory structure same as that of #3374 * Make directory structure similar to #3374 * add new files to test file script exemptions and nit fixes * diable ktlint max-line-length * disable ktlint-max-line * disable ktlint max-length * remove testonly attribute from tests * add new files to script exemptions * nit fix * nit fix * nit fix * Apply review suggestions * ktlint fix * nit fix * Apply nit fixes * Apply nit fixes * Remove script constants * nit fix * add todo * add todo * nit fixes * add test * add testOnly * nit fixes * nit fix * nit fixes * nit fixes * nit fixes * nit fixes * fix textproto file * add exemptions to textproto * sort failures lexicographically * add new files to exemption list * nit fixes * nit fixes * add kdocs * nit fix * nit fix * Trigger CI failure exclusively * Revert CI check failure * Fix #3340: Create a script to ensure KDoc presence * Add initial tests * Add more tests and add documentation * nit fix * nit fix * nit fix * nit fix * Add more files to kdoc exemptions * Implementation part 1 * add exemption proto for todos * nit fixes * delete open issues * nit fixes * nit fixes * Add redundant exemptions check * rectify static_checks * Remove duplicate proto libraries * nit fixes * nit fixes and add redundant exemptions check to script * nit fixes * add activity to exemption * nit fixes * nit fixes * nit fixes * nit fixes * revamption part 1 * nit fixes * update branch * Revamped approach * nit fixes * add kdocs * delete open_issues.json * Fix #3317: Update old todos * nit fixes * Remove resolved todos * Repurpose todos stage 1 * Repurpose todos stage 2 * Fix #3318: Add check to ensure all todos of closed issue are resolved * nit fixes * updating todos part 3 * add tests * update more todos * add network module * update todo no. 322 * correct if condition in workflow * Remove unused imports * Add duplicate comment check condition * nit fixes * Correct exit code * Revert "Remove unused imports" This reverts commit 65ee6e9. * Revert "update todo no. 322" This reverts commit c150cdc. * Repurpose no. 322 * nit fixes * replace issue number for backend model todo * add tests * link new filed issue * nit fixes * add exemptions * nit fixes * nit fixes * nit fixes * nit fixes * nit fixes * add remaining tests for all cases * nit fixes * nit fixes * nit fixes Co-authored-by: Sparsh Agrawal <sparsh@Sparshs-MacBook-Air.local>

* Make nit suggestions * Make nit suggestions * Fix #3300: Add check for accessibility labels for activities * Add CI check for label presence for activities * Add ScriptConstants file to exemptions test list * Test if: always() * Revert "Test if: always()" This reverts commit b72e419 * Add if: always() * Add if: always() * Add if: always() * Apply review suggestions on PR * Make review suggestions from regex pattern checks * Implement review suggestions based on #3340 * Refactor PR as per feedback recieved * Implement feedback suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * Implement review suggestions * nits * Nit fixes * Nit fixes * Nit fixes * Implement feedback suggestions as per #3340 * nit fix * nit fix * nit fix * nit fix * Review suggestions part 1 * Implement review suggestions part 2 * nit fixes * update static_checks * add test to repository file * nit fix * nit fix * nit fix * Do review suggestions * Implement review suggestions * Implement review suggestions * nit fix * Implement review suggestions * Add Ben as a codeowner for the ScriptExemptions file * bazel files nit fix * Introduce a library for the regex assets * Make directory structure same as that of #3374 * Make directory structure similar to #3374 * add new files to test file script exemptions and nit fixes * diable ktlint max-line-length * disable ktlint-max-line * disable ktlint max-length * remove testonly attribute from tests * add new files to script exemptions * nit fix * nit fix * nit fix * Apply review suggestions * ktlint fix * nit fix * Apply nit fixes * Apply nit fixes * Remove script constants * nit fix * add todo * add todo * nit fixes * add test * add testOnly * nit fixes * nit fix * nit fixes * nit fixes * nit fixes * nit fixes * fix textproto file * add exemptions to textproto * sort failures lexicographically * add new files to exemption list * nit fixes * nit fixes * add kdocs * nit fix * nit fix * Trigger CI failure exclusively * Revert CI check failure * Fix #3340: Create a script to ensure KDoc presence * Add initial tests * Add more tests and add documentation * nit fix * nit fix * nit fix * nit fix * Add more files to kdoc exemptions * Implementation part 1 * add exemption proto for todos * nit fixes * delete open issues * nit fixes * nit fixes * Add redundant exemptions check * rectify static_checks * Remove duplicate proto libraries * nit fixes * nit fixes and add redundant exemptions check to script * nit fixes * add activity to exemption * nit fixes * nit fixes * nit fixes * nit fixes * revamption part 1 * nit fixes * update branch * Revamped approach * nit fixes * add kdocs * delete open_issues.json * Fix #3317: Update old todos * nit fixes * Remove resolved todos * Repurpose todos stage 1 * Repurpose todos stage 2 * Fix #3318: Add check to ensure all todos of closed issue are resolved * nit fixes * updating todos part 3 * add tests * update more todos * add network module * update todo no. 322 * correct if condition in workflow * Remove unused imports * Add duplicate comment check condition * nit fixes * Correct exit code * Revert "Remove unused imports" This reverts commit 65ee6e9. * Revert "update todo no. 322" This reverts commit c150cdc. * Repurpose no. 322 * nit fixes * replace issue number for backend model todo * add tests * link new filed issue * nit fixes * add exemptions * nit fixes * nit fixes * nit fixes * nit fixes * nit fixes * add remaining tests for all cases * point error messages to wiki * nit fixes * nit fixes * nit fixes * nit fixes * log check medication only once * reference exact urls * correct flaky test Co-authored-by: Sparsh Agrawal <sparsh@Sparshs-MacBook-Air.local>

BenHenning added 3 commits June 25, 2021 19:29

Migrate compute affected tests script to Kotlin.

c3f8d78

No tests for the script yet.

Add BUILD.bazel docstrings.

95a389d

Enable linters for scripts code. Fix ktlint issues.

5465f11

BenHenning assigned Sparsh1212 Jun 26, 2021

Sparsh1212 mentioned this pull request Jun 26, 2021

[DO NOT MERGE] Test compute affected tests kotlin script by regex checks #3375

Closed

8 tasks

Sparsh1212 assigned BenHenning and unassigned Sparsh1212 Jun 26, 2021

Sparsh1212 mentioned this pull request Jun 27, 2021

Fix #3291: Add check for XML syntax validation #3341

Merged

8 tasks

This was referenced Jun 28, 2021

[DO NOT MERGE] Fix #3285: Create script to compile list of maven dependencies [WIP] #3338

Closed

Fix #3290: Add support for generic regex pattern matching #3340

Merged

Add tests for the compute affected tests script.

d5117b6

Fix some typos & realized bug in the script.

Check out develop first so that exists in CI.

1304943

See comment in PR for context.

BenHenning added 5 commits June 30, 2021 17:48

Reorganize & refactor, and improve tests.

33528f7

This splits up the test & script into multiple utility packages (for common & testing utilities), reorganizes the test to be in a proper javatests/ & java/ arrangement, and adds tests for checking the error conditions of the test.

Add tests for executeCommand().

8c578bd

This also refactors that function to be part of a class so that its operating conditions are configurable (namely the process timeout).

Add tests for BazelClient.

f965e5a

Add tests for GitClient.

a56f31c

Add tests for TestBazelWorkspace & documentation.

f830c67

Also, fix a few small issues in the utility.

BenHenning mentioned this pull request Jul 2, 2021

Fix #3292: Add check for test files presence for prod files #3343

Merged

8 tasks

BenHenning added 4 commits July 1, 2021 22:13

Add tests git TestGitRepository.

9ec63b1

Merge branch 'develop' into migrate-compute-affected-tests-to-kotlin

1841f20

Add support for changing relative develop branch.

25f0505

Also, fix some changed behaviors that broke the affected tests script.

Fix linter error found in CI.

877337c

Remove unnecessary extra checkout from CI.

4a32f2b

BenHenning mentioned this pull request Jul 3, 2021

Verify Bazel CI test changes are working as expected #2691

Open

8 tasks

Disable test that can't pass in CI.

6425f75

See #2691 for context.

BenHenning assigned anandwana001 and Sparsh1212 Jul 3, 2021

vinitamurthi approved these changes Jul 3, 2021

View reviewed changes

vinitamurthi removed their assignment Jul 3, 2021

Sparsh1212 approved these changes Jul 3, 2021

View reviewed changes

oppiabot bot unassigned Sparsh1212 Jul 3, 2021

anandwana001 approved these changes Jul 3, 2021

View reviewed changes

anandwana001 merged commit 52d6ca2 into develop Jul 3, 2021

anandwana001 deleted the migrate-compute-affected-tests-to-kotlin branch July 3, 2021 07:06

BenHenning mentioned this pull request Jul 6, 2021

Fix part of #2691: Mitigate affected tests script CI hang problem #3416

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix part of #2691: Migrate compute affected tests script to Kotlin #3374

Fix part of #2691: Migrate compute affected tests script to Kotlin #3374

BenHenning commented Jun 26, 2021 •

edited

BenHenning commented Jun 26, 2021

Sparsh1212 commented Jun 26, 2021

Sparsh1212 commented Jun 26, 2021

BenHenning commented Jun 26, 2021

Sparsh1212 commented Jun 26, 2021

BenHenning commented Jun 30, 2021

BenHenning commented Jun 30, 2021

BenHenning commented Jun 30, 2021

BenHenning commented Jul 3, 2021

BenHenning commented Jul 3, 2021

BenHenning commented Jul 3, 2021

BenHenning commented Jul 3, 2021 •

edited

vinitamurthi left a comment

Sparsh1212 commented Jul 3, 2021

oppiabot bot commented Jul 3, 2021

anandwana001 left a comment

BenHenning commented Jul 3, 2021

Fix part of #2691: Migrate compute affected tests script to Kotlin #3374

Fix part of #2691: Migrate compute affected tests script to Kotlin #3374

Conversation

BenHenning commented Jun 26, 2021 • edited

Explanation

Checklist

BenHenning commented Jun 26, 2021

Sparsh1212 commented Jun 26, 2021

Sparsh1212 commented Jun 26, 2021

BenHenning commented Jun 26, 2021

Sparsh1212 commented Jun 26, 2021

BenHenning commented Jun 30, 2021

BenHenning commented Jun 30, 2021

BenHenning commented Jun 30, 2021

BenHenning commented Jul 3, 2021

BenHenning commented Jul 3, 2021

BenHenning commented Jul 3, 2021

BenHenning commented Jul 3, 2021 • edited

vinitamurthi left a comment

Choose a reason for hiding this comment

Sparsh1212 commented Jul 3, 2021

oppiabot bot commented Jul 3, 2021

anandwana001 left a comment

Choose a reason for hiding this comment

BenHenning commented Jul 3, 2021

BenHenning commented Jun 26, 2021 •

edited

BenHenning commented Jul 3, 2021 •

edited