Spec Speedup: Use Knapsack to Run Parallel Builds #8390

mstruve · 2020-06-10T17:07:30Z

What type of PR is this? (check all applicable)

Optimization

Description

This PR introduces a service called Knapsack to help us parallelize our spec suite as evenly as possible. The first time I ran Knapsack in our build it ran every test separately and recorded the time it took to run the test. Using this information Knapsack then splits the tests up for us into 3 equally timed groups to run in parallel each time we run our test suite. This is how regular mode works.

The changes in this PR are introducing the gem and setting it up. All of them were made with the help of the Knapsack installation guide which walks you through all the changes you should make to get it working properly.

NOTE - There is currently a bug with the parallelization in Travis that causes the --local flag for our bundler command to be ignored. This means on your first build, since there is no travis cache, the jobs will likely take 13min. I am in contact with Travis support to get this resolved.

Why aren't we using Queue Mode? Ideally, we want to use queue mode. In Queue Mode Knapsack sends us groups of 3-5 specs at once and then when they finish, sends another group of specs. It keeps doing this until all specs have been run. This is obviously the fastest approach but we ran into some errors with the jobs hanging. My goal is to get the regular version out then try to debug that hanging issue so we can use queue mode.

How much does Knapsack cost? FREE bc we are open source and I must say the founder has been extremely helpful in getting us going and holding my hand through the integration process.

@snackattas

mstruve · 2020-06-10T19:37:54Z

.travis.yml

 before_script:
  - bundle exec rails db:create
 script:
  - bundle exec rails webpacker:compile
  - bundle exec rails db:schema:load
  - './cc-test-reporter before-build'
-  - 'npx percy exec -- bundle exec rspec spec --color --tty --format progress'
+  - 'bundle exec rake knapsack_pro:rspec'


It is possible to setup Percy with KnapsackPro but I choose to forego that since we are not using Percy at the moment and wanted to keep this all as simple as possible

Just wondering, does using knapsack mean that we'll get 4 csv files sent up to s3 (3 for each knapsack run and 1 for the rss spec run) on master? (thats perfectly fine, just curious).

mstruve · 2020-06-10T19:38:17Z

.travis.yml

-  - 'bundle exec bundle-audit check --update --ignore CVE-2015-9284'
-  - yarn build-storybook
-  - bundle exec rails runner -e production 'puts "App booted successfully"'
+  - if [ "$KNAPSACK_PRO_CI_NODE_INDEX" == "0" ]; then yarn test --colors; fi


Split up other tasks to run on individual nodes

mstruve · 2020-06-10T19:39:49Z

.travis.yml

@@ -33,6 +33,14 @@ env:
    - SECRET_KEY_BASE=dummydummydummy
    - GITHUB_KEY=dummy
    - GITHUB_SECRET=dummy
+    - KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true
+    - KNAPSACK_PRO_LOG_LEVEL=info
+    - KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN="spec/services/rss_reader_spec.rb"


The RSS reader spec for some reason had a tendency to cause jobs to hang. I think it has to do with how we use the VCR so I choose to remove it from the KnapsackPro specs and run it by itself for now so we could get this out. I plan on looping back to figure out exactly what the issue is there.

Big 👍 on not letting a single spec file block us!

mstruve · 2020-06-10T19:41:03Z

spec/requests/api/v0/articles_spec.rb

@@ -442,8 +442,8 @@
  end

  describe "POST /api/articles" do
-    let_it_be(:api_secret) { create(:api_secret) }
-    let_it_be(:user) { api_secret.user }
+    let(:api_secret) { create(:api_secret) }


These specs popped up as flaky when I started running KnapsackPro builds bc of data issues so I changed to using non-persistent objects. I think we should get in the habit of this so eventually we can remove our dependency on order.

agreed! let it be causes trouble

mstruve · 2020-06-10T19:41:39Z

.rspec

@@ -1,2 +1,3 @@
 --color
+--format progress


Move this flag here so we still get the progress with KnapsackPro

Since the order specs are run in isn't fixed, might be worth considering using --format documentation, so one could see the order at a glance (and debug issues if order happens to be a factor in flakiness). Thats my 2 cents.

knapsack_pro in Regular Mode shows at the beginning of output something like

rspec --default-path spec "spec/system/articles/user_visits_articles_by_timeframe_spec.rb" ...

You can copy-paste to run in the same order tests.

In Queue Mode there will be more examples in output for each given set of tests fetched from Queue. Thanks to that you can run tests in development in the same order.

If you use RSpec seed then you can pass additionally --seed to retry in exact order https://knapsackpro.com/faq/question/how-to-find-seed-in-rspec-output-when-i-use-queue-mode-for-rspec

cool, I see it now, thanks! no --format documentation needed

one day I think it'd be nice to use --order random but I don't think we're quite there yet/that could probably be a separate piece of work to get working

mstruve · 2020-06-10T19:43:35Z

.travis.yml

@@ -79,7 +86,7 @@ deploy:
    condition: $TRAVIS_COMMIT_MESSAGE =~ \[deploy\]
 notifications:
  slack:
-    if: branch = master
+    if: branch = master && "$KNAPSACK_PRO_CI_NODE_INDEX" == "0"


We only want a single deploy to happen and slack notification too I guess lol

I just realized this is for the slack notification, not the actual deploy 😬

Zhao-Andy

LGTM! I don't know much about the setup but the build passed and I AM VERY EXCITED FOR THIS!!!

Zhao-Andy · 2020-06-10T19:48:45Z

Oh just noticed our CodeClimate coverage went down by a ton. Wonder if that's configurable?

mstruve · 2020-06-10T19:51:17Z

~~@Zhao-Andy I have had this issue on other PRs where I remove code or tweak config files and codeclimate reports a huge drop in coverage. I am not sure why it does it but in the past I just ignore it.~~ Of course that was before I was messing with the actual travis yml file and calculating that coverage. Taking a closer look at this

maestromac · 2020-06-10T21:40:52Z

.travis.yml

+  - if [ "$KNAPSACK_PRO_CI_NODE_INDEX" == "2" ]; then bundle exec rspec spec/services/rss_reader_spec.rb; fi
+after_script:
+  - ./cc-test-reporter format-coverage -t simplecov -o ./coverage/codeclimate.$KNAPSACK_PRO_CI_NODE_INDEX.json ./coverage/spec/.resultset.json
+  - ./cc-test-reporter sum-coverage --output - --parts $KNAPSACK_PRO_CI_NODE_INDEX coverage/codeclimate.*.json | ./cc-test-reporter upload-coverage --input -


What does the standalone - do here?

Based on the coverage result on the pr check and on codeclimate, I don't think the sum-coverage here is properly combining all the results. Definitely not a blocker though, we can figure this out afterward.

This change isn't sending lcov's result

missing for the first node
'./cc-test-reporter format-coverage -t lcov -o coverage/codeclimate.lcov.json'

This I actually stole from @citizen428's parallel travis PR #6645 and the purpose of it is to track the coverage for all the nodes then combine them at the end. However, the code coverage does make it seem like that is not the case... 🤔 Given the % I feel like we got 2 nodes and not 3

Definitely need the total nodes not the node index when combining them 😅

You may find some tips about CodeClimate setup here
https://knapsackpro.com/faq/question/how-to-use-codeclimate-with-knapsack_pro
there are 2 articles but for different CI providers than Travis. But still, you may find some inspiration there.

If you will need at some point in the future setup for simplecov and knapsack_pro Queue Mode then please check this https://knapsackpro.com/faq/question/how-to-use-simplecov-in-queue-mode

What does the standalone - do here?

@maestromac It's a shell thing for representing standard input or output, see e.g. https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html

Guideline 13:
For utilities that use operands to represent files to be opened for either reading or writing, the '-' operand should be used to mean only standard input (or standard output when it is clear from context that an output file is being specified) or a file named -.

ArturT · 2020-06-10T22:39:05Z

.travis.yml

@@ -33,6 +33,14 @@ env:
    - SECRET_KEY_BASE=dummydummydummy
    - GITHUB_KEY=dummy
    - GITHUB_SECRET=dummy
+    - KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true


KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true flag is for knapsack_pro Queue Mode. But let's keep it here so you won't forget to add it when you will try Queue Mode.

Here is more explanation of why it's needed for Travis CI
https://github.com/KnapsackPro/knapsack_pro-ruby#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node
Basically allows us to remember a split of tests and when you retry only a single failed parallel job then a proper set of tests will be run on retried Travis job.

Oh, great to know!!! Yeah I figured I would keep it in for now since queue mode is our goal

ArturT · 2020-06-10T22:46:04Z

Hi all, if you would like to learn more about difference between knapsack_pro Regular Mode and Queue Mode, here you find an article and videos about it
https://docs.knapsackpro.com/2020/how-to-speed-up-ruby-and-javascript-tests-with-ci-parallelisation

@mstruve You can also add a badge to the readme file for a knapsack pro. You can find it next to API token in https://knapsackpro.com/dashboard

There is also option to make the Knapsack Pro dashboard public for given API token, just edit API token and enable public dashboard. So when someone clicks on badge then can access dashboard with recorded tests. I added this option a few months ago for open source projects so anyone can access recorded tests.

mstruve · 2020-06-11T00:43:57Z

.travis.yml

+  - ./cc-test-reporter sum-coverage --output - --parts $COVERAGE_REPORTS_TOTAL coverage/codeclimate.*.json | ./cc-test-reporter upload-coverage --input -
+jobs:
+  include:
+    - stage: Deploy


Ensures that all specs pass before we kick off a job to deploy the code. Reference: https://docs.travis-ci.com/user/build-stages/deploy-heroku/

The problem with this right now is that it runs the install steps still, going to work on moving those into the script step for tests.

citizen428 · 2020-06-11T02:27:08Z

.travis.yml

@@ -33,6 +33,14 @@ env:
    - SECRET_KEY_BASE=dummydummydummy
    - GITHUB_KEY=dummy
    - GITHUB_SECRET=dummy
+    - KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true
+    - KNAPSACK_PRO_LOG_LEVEL=info
+    - KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN="spec/services/rss_reader_spec.rb"


Big 👍 on not letting a single spec file block us!

citizen428 · 2020-06-11T03:05:49Z

.travis.yml

+          on_success: change
+          on_failure: always
+          rooms:
+            secure: vzIee4jDgPSRY4szZPdD/jW7YW4GzGqo5NoLV9Exz9TBoWH9UqJnc0TOb2YN84Ys5baRK7LOqxpfp8kFveZkrKGi7/ypeEJlpc9E5UqVh/bwQhvOGrKEg1fvNXbARRnO/sJ49o1CMvroMWvt3GurzuuY9Qu2r+3NBjn9aVwLnLzXsBuF+m2lLoeSkHnW13OC73EeJMsse6JBoCe3gp/srDwISp9+MU+sEAPaY333WK9Vk1kdG7D5oUIuT7743airLRiyWiNUCD1450g864628CEOEZKJAAtqk6kTmvwB91DJMnhD/XhMm4H21kd54YHy0fhqzcG8hYd1lDZuUfrOBfpdEtfnpcRwMyMpY+FPPHXkHhck3OiLJnzkV4L+Lr5W/RvDJ63Ye2nxT4hOItLWaoZWax/LhoIrhZjgYBc4JhiGRQJ8m2HzoRyceeG9Y80vayGVN7y46sjYHP5NHRI36qmJipneDRAJklBTXLdYATvVM/6Mh9B7+H/nBGR6UVJLBC/txi2C8rZRjKBZ/i9e+q/MZs0UEvOuvbz9BXKU08rI+rarJqH3h5Ji9G/k3M0mQ8EfvadabA9lu+gNUAAnq+vwLETweKvfbRpDQjVBKnWsOJoUl9aarfkBn3lhQE8fxZJT/GchLGZPx/CWUE4o1OhliBA9avJ7WINyYStM4Mc=


Not sure how sensitive this is, just double checking if it might make sense to put this into a Travis env var and read it from there?

This is an encrypted var generated based on their official instruction. It shouldn't be a problem but I can extract it on a different PR

@maestromac If you don't think it's problematic, save yourself the work. I was just checking.

rhymes

Looks great!

Just one question: are the JS tests run three times?

maestromac

👏 👏 👏 👏 this is going to save all of us so much time!

mstruve · 2020-06-11T15:25:19Z

@rhymes nope! Thanks to a few if statements here https://github.com/thepracticaldev/dev.to/pull/8390/files#diff-354f30a63fb0907d4ad57269548329e3R70 it will only run on node 0 while other tasks will run on the other nodes.

- if [ "$KNAPSACK_PRO_CI_NODE_INDEX" == "0" ]; then yarn test --colors; fi

rhymes · 2020-06-11T15:27:13Z

LGTM! @mstruve :D

* Fix erblint errors on the `app/views/listings` folder (#8374) * Fix erblint errors on the `app/views/listings` folder * Move `to_json()` methods to the controller * Update UserDecorator not to cache AR Object (#8378) * Update UserDecorator not to cache AR Object * Change cache name * Change cache name to reduce invalidation frequency * Add json template to Pages (#8357) * Add json template to pages * Add jsonb field, tweak form * Spec Cleanup: Remove Unused and Redundant Code from Specs (#8385) * [deploy] Add crayons to reserved words (#8386) * Fix erblint errors on the `app/views/internal` folder (#8387) * Fix a11y issues flagged by axe on homepage (#8380) * Fix a11y issues flagged by axe on homepage This commit brings the number of violations found on homepage (local, not prod) down from ~100 to ~10. It should have no serious impact on the visual layout or design of the page. * Change aria-labels to be consistent * Flaky Spec Fix: Clear Elasticsearch Data For js: true Specs to Fix Server Errors (#8388) Co-authored-by: Vaidehi Joshi <vaidehi.sj@gmail.com> * [deploy] Bump @percy/agent from 0.26.8 to 0.26.9 (#8325) Bumps [@percy/agent](https://github.com/percy/percy-agent) from 0.26.8 to 0.26.9. - [Release notes](https://github.com/percy/percy-agent/releases) - [Changelog](https://github.com/percy/percy-agent/blob/master/CHANGELOG.md) - [Commits](percy/percy-agent@v0.26.8...v0.26.9) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com> * [deploy] Refactor 🚀 : Replaced Chat Channel Setting page with Preact component. (#8271) * Feature 🚀 : Ability to delete messages in chat channels - Sending message ID to frontend - Deleting Message - Use pusher to delete message realtime * Minor Bug 🐞: Show message action only for current user - User can delete or edit their own messages * Test cases added * Bug 🐞: Update message id for receiver Message id was not sent to receiver by pusher * Refactoring🛠: Message controller refactoring * Test Cases📝 : Specs for Delete message added * Feature 🚀 : Ability to edit messages * Test Cases📝 : Specs for Edit message added * 🐞 Channel List bug in mobile view * changes in joining request responses * 🛠 Request Managers Frontend ready * changes in routes for adding and removing remembership * 🚀 New routes implemented for request manager * fix issues * changes in api for remove membership * Merge conflict resolved * 🐞 Channel List bug in mobile view * changes in joining request responses * 🛠 Request Managers Frontend ready * changes in routes for adding and removing remembership * 🚀 New routes implemented for request manager * fix issues * changes in api for remove membership * 🛠Optimizing for CodeClimate * 🛠Optimizing again for CodeClimate * 🛠Optimizing again 2 for CodeClimate * 🛠 Added more test cases * 🛠 Optimizing code * 🚀 Action to open setting page added * 🛠 Dummy page added for chat setting * add JSON routes for chat channel settings * Integrate chat channel settings API with UI * 🐞 Channel List bug in mobile view * 🛠 Request Managers Frontend ready * changes in routes for adding and removing remembership * 🚀 New routes implemented for request manager * fix issues * 🐞 Channel List bug in mobile view * 🛠 Request Managers Frontend ready * changes in routes for adding and removing remembership * 🚀 New routes implemented for request manager * fix issues * 🛠Optimizing again for CodeClimate * 🚀 Action to open setting page added * 🛠 Dummy page added for chat setting * add JSON routes for chat channel settings * Integrate chat channel settings API with UI * Fix PR requested changes * Add JSDoc documentation to exported functions * fix/add rspec test cases * refactor channelSettinngs render part * add test cases for chat channel settings component * fix code-climate * add rest component tests * add more test coverage * fix code-climate bugs * add API function test Co-authored-by: Sarthak Sharma <7lovesharma7@gmail.com> Co-authored-by: Parasgr-code <paras.gaur@skynox.tech> * Fix erblint errors on the `app/views/users` folder (#8389) * Fix erblint errors on the `app/views/users` folder * Fix mistake * Properly return json (#8395) [deploy] * Implement chronological sorting of search results (#8193) * Implement search ordering by allowing sort_by and sort_direction params to be passed through from the search form/page. * Change ID of select to 'sort' instead of 'order' for consistency * Fix styling and data attributes of select options * Enable front end to pass sort_by and sort_direction params for searches to the back end * Ensure that only query string params that are supposed to be changed are changed in each function * Respond to review comments (additional commits to come but these were the quick changes) * Move options for select into template * Use addEventListener instead of using onchange in response to review comments * Add tabs for sorting search results * Finish implementing tabs for ordering search results * Remove CSS rule applying to element that no longer exists * Remove more rules created for an element that has been removed * Remove more code added for the select dropdown that I've removed * Remove duplicate jobs banner * Update example code (#8397) * Fix erblint errors on the `app/views/pages` folder (#8401) * Fix erblint errors on the `app/views/layouts` folder (#8400) * Allow users to dismiss announcements (#8396) [deploy] * Provide default crayon styles when creating banners * Import crayons styles into /internal pages. * Allow admins to use crayons banner styles when creating broadcasts. * Preview banner styles with crayons. * Add a "x" close button to announcements, fix styling to accommodate it * Use constant and helper for broadcast banner styles * Add VALID_BANNER_STYLES frozen constant to Broadcast class. * Add banner_style helper for determining banner style class. * Clean up preview CSS, import broadcast styles into preview * Add close button click handler, clean up initializeBroadcast * Add close functionality to announcement banner Also hide announcement if the user has explicitly seen and "X"-ed out of it. * Add system specs around rendering + dismissing broadcasts * Add some (truly beautiful) JS documentation * Add visible class to broadcast when previewed from internal * [deploy] Spec Speedup: Use Knapsack to Run Parallel Builds (#8390) * Fix erblint errors on the `app/views/doorkeeper` folder (#8399) * Fix erblint errors on the `app/views/mailers` folder (#8402) * Allow Knapsack to work for Forks (#8413) * Add QA instructions to pull request template (#8412) * Bump top-bar's z-index to allow dropdown to render with announcement (#8415) * Fix remaining erblint errors (#8405) * Fix remaining erblint errors * Fix last error * Fix erblint errors on the `app/views/liquids` folder (#8403) * Fix erblint errors on the `app/views/liquids` folder * Make tests pass * Make failing test pass * [deploy] Update GithubReposController to clean up inaccessible repositories (#8382) * Update Users::Sidebar not to cache AR object (#8394) * [deploy] Update ReactionsController not to cache AR Object (#8393) * Hide site-wide messaging for opted-out users (#8409) [deploy] Co-authored-by: Alberto Pérez de Rada Fiol <apdrf.94@gmail.com> Co-authored-by: Mac Siri <krairit.siri@gmail.com> Co-authored-by: Josh Puetz <josh@dev.to> Co-authored-by: Molly Struve <mollylbs@gmail.com> Co-authored-by: Andy Zhao <17884966+Zhao-Andy@users.noreply.github.com> Co-authored-by: Jacob Herrington <jacobherringtondeveloper@gmail.com> Co-authored-by: Vaidehi Joshi <vaidehi.sj@gmail.com> Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com> Co-authored-by: narender2031 <narender2031@gmail.com> Co-authored-by: Sarthak Sharma <7lovesharma7@gmail.com> Co-authored-by: Parasgr-code <paras.gaur@skynox.tech> Co-authored-by: Dana Scheider <dana.scheider@gmail.com> Co-authored-by: Michael Kohl <citizen428@gmail.com>

…orem#8390)" This reverts commit f647944.

pr-triage bot added the PR: draft bot applied label for PR's that are a work in progress label Jun 10, 2020

mstruve closed this Jun 10, 2020

mstruve reopened this Jun 10, 2020

mstruve force-pushed the mstruve/knapsack branch from 1a8aaee to 8d44430 Compare June 10, 2020 18:31

Spec Speedup: Use Knapsack to Run Parallel Builds

6e8981f

mstruve force-pushed the mstruve/knapsack branch from 8d44430 to 6e8981f Compare June 10, 2020 19:14

Add Knapsack to CI test README

4b8fea6

mstruve commented Jun 10, 2020

View reviewed changes

mstruve marked this pull request as ready for review June 10, 2020 19:43

mstruve requested review from jacobherrington and a team as code owners June 10, 2020 19:43

pr-triage bot added PR: unreviewed bot applied label for PR's with no review and removed PR: draft bot applied label for PR's that are a work in progress labels Jun 10, 2020

mstruve commented Jun 10, 2020

View reviewed changes

mstruve requested review from maestromac and rhymes June 10, 2020 19:43

Zhao-Andy approved these changes Jun 10, 2020

View reviewed changes

pr-triage bot added PR: partially-approved bot applied label for PR's where a single reviewer approves changes and removed PR: unreviewed bot applied label for PR's with no review labels Jun 10, 2020

mstruve mentioned this pull request Jun 10, 2020

Debug Travis CI parallel jobs hangs/freeze when using knapsack_pro gem for tests parallelism #8274

Closed

11 tasks

maestromac reviewed Jun 10, 2020

View reviewed changes

benhalpern approved these changes Jun 10, 2020

View reviewed changes

mstruve requested a review from citizen428 June 10, 2020 21:48

We want the TOTAL nodes not the node index

b334259

pr-triage bot added PR: unreviewed bot applied label for PR's with no review and removed PR: partially-approved bot applied label for PR's where a single reviewer approves changes labels Jun 10, 2020

Add back accidently removed Preact coverage report

d53ac29

ArturT reviewed Jun 10, 2020

View reviewed changes

We need to upload a total of 4 coverage reports

5eed161

Run deploy as a separate job after all test jobs have completed

af27e15

mstruve commented Jun 11, 2020

View reviewed changes

mstruve added 4 commits June 10, 2020 20:00

move install steps to script so they are only executed for tests

029083d

keep coverage reports in after_script

9482bbd

yml is hard

94801b5

skip after_script as well

70375a9

citizen428 reviewed Jun 11, 2020

View reviewed changes

rhymes approved these changes Jun 11, 2020

View reviewed changes

pr-triage bot added PR: partially-approved bot applied label for PR's where a single reviewer approves changes and removed PR: unreviewed bot applied label for PR's with no review labels Jun 11, 2020

maestromac approved these changes Jun 11, 2020

View reviewed changes

mstruve merged commit f647944 into master Jun 11, 2020

mstruve deleted the mstruve/knapsack branch June 11, 2020 15:27

pr-triage bot added PR: merged bot applied label for PR's that are merged and removed PR: partially-approved bot applied label for PR's where a single reviewer approves changes labels Jun 11, 2020

pr-triage bot added PR: partially-approved bot applied label for PR's where a single reviewer approves changes and removed PR: merged bot applied label for PR's that are merged labels Jun 12, 2020

citizen428 pushed a commit that referenced this pull request Jun 15, 2020

[deploy] Spec Speedup: Use Knapsack to Run Parallel Builds (#8390)

548a54f

atsmith813 added a commit to atsmith813/dev.to that referenced this pull request Jun 16, 2020

Revert "[deploy] Spec Speedup: Use Knapsack to Run Parallel Builds (f…

1fc43b4

…orem#8390)" This reverts commit f647944.

mstruve mentioned this pull request Jun 23, 2020

Spec Fix: Use Truncation For Tests Where Transactions Get Stuck #8865

Merged

1 task

Spec Speedup: Use Knapsack to Run Parallel Builds #8390

Spec Speedup: Use Knapsack to Run Parallel Builds #8390

Uh oh!

Conversation

mstruve commented Jun 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this? (check all applicable)

Description

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArturT Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mstruve Jun 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Zhao-Andy left a comment

Choose a reason for hiding this comment

Uh oh!

Zhao-Andy commented Jun 10, 2020

Uh oh!

mstruve commented Jun 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maestromac Jun 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mstruve Jun 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

citizen428 Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mstruve Jun 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArturT commented Jun 10, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

mstruve commented Jun 10, 2020 •

edited

Loading

ArturT Jun 11, 2020 •

edited

Loading

mstruve Jun 10, 2020 •

edited

Loading

mstruve commented Jun 10, 2020 •

edited

Loading

maestromac Jun 10, 2020 •

edited

Loading

mstruve Jun 10, 2020 •

edited

Loading

citizen428 Jun 12, 2020 •

edited

Loading

mstruve Jun 10, 2020 •

edited

Loading

rhymes left a comment •

edited

Loading