Added GCP JMH Benchmark Workflow

[armughan11]

Adds a workflow that is triggered via a "/benchmark" comment on a pull request. Clones the main and PR branch on the GCP instance and runs the ./gradlew jmh on each one of them, returning the results as a comment on the pull request. Running on my test instance for review, will change the credentials accordingly once approved.

[armughan11]

Updated and tested after final changes and added VM start and stop steps.

[msridhar]

Looking great! Just a couple of final comments / questions

[msridhar]

Why do we need this cd $BRANCH_NAME/ command in this script and also run_pr_benchmarks.sh?

[armughan11]

I just kept this for the sake of consistency across the two files. I am running the PR benchmark on the $BRANCH_NAME/pr/NullAway folder (the whole $BRANCH_NAME directory was set up to avoid errors due to any residual files from other PRs), and I am navigating back to the root directory after that's done. I can omit the cd $BRANCH_NAME from main, but that would mean that I will have to go up two directories to the $BRANCH_NAME folder in the step where I run the gcloud ssh command. Happy to make those changes if they are more suitable @msridhar. Thank you!

[msridhar]

Ok, it's fine to keep, was just curious.

[msridhar]

I think this is good to go! Will merge once we do some more testing. Thanks again @armughan11!

[msridhar]

Ok, it's fine to keep, was just curious.

[msridhar]

Update: I successfully tested this change on my fork!

https://github.com/msridhar/NullAway/actions/runs/5580019520/jobs/10196359492
msridhar#10 (comment)

This was with a PR that deliberately introduced a slowdown. I did a couple of tweaks in naming based on that.

@armughan11 I have two more comments / questions for you.

1. Benchmarking can take a while, and there is no feedback on the PR that anything is happening until the benchmarks complete. I was wondering, as a first thing could the workflow add a thumbs up to the /benchmark comment, to show that it is running? It seems possible, e.g., using this action (see here).
2. Regarding the step to stop the VM running with always(), just curious, will this step run even if the job is canceled?

[armughan11]

Thanks for the update @msridhar!

1. This should be doable, I will look into it. Ideally, I would add this step right after the GCP authentication is complete or right at the beginning, but if you want the reaction after a specific step, kindly let me know, so I can make the changes accordingly.
2. Yes! That's why you would find the cleanup step with always() as well, which ensures that we don't end up with cloning errors for subsequent runs due to residual files. In short, steps marked always() aren't cancellable.

[msridhar]

1. This should be doable, I will look into it. Ideally, I would add this step right after the GCP authentication is complete or right at the beginning, but if you want the reaction after a specific step, kindly let me know, so I can make the changes accordingly.

I was just thinking right at the beginning, to indicate something is running.

2. Yes! That's why you would find the cleanup step with always() as well, which ensures that we don't end up with cloning errors for subsequent runs due to residual files. In short, steps marked always() aren't cancellable.

👍

[armughan11]

Just pushed a change. This should add the reaction once the job starts! @msridhar

[msridhar]

Confirmed that adding the reaction works: msridhar#10 (comment)

I think this is good to go. I'm going to land it and test on the main NullAway repo 🙂

[msridhar]

Thanks again for the great contribution @armughan11!