Add Azure Block Blob Storage backend by c-w · Pull Request #4685 · celery/celery

c-w · 2018-04-27T17:45:07Z

Description

This pull request adds a new results backend. The backend is implemented on top of the azure-storage library which uses Azure Blob Storage for a scalable low-cost PaaS backend.

The backend was load tested via a simple nginx/gunicorn/sanic app hosted on a DS4 virtual machine (4 vCores, 16 GB RAM) and was able to handle 600+ concurrent users at ~170 RPS.

The commit also contains a live end-to-end test to facilitate verification of the backend functionality. The test is activated by setting the AZUREBLOCKBLOB_URL environment variable to azureblockblob://{ConnectionString} where the value for ConnectionString can be found in the Access Keys pane of a Storage Account resources in the Azure Portal:

This pull request was created together with @ankurokok, @dkisselev, @evandropaula, @martinpeck and @michaelperel.

auvipy

plz check the lint error, fixing those will trigger test build

c-w · 2018-04-27T18:42:48Z

Thanks for the reply, @auvipy. The lint steps should now pass.

I needed to make one change to be able to run the configcheck lint step (TOXENV=configcheck tox) in the docker container, so if you wish I can make a follow-up pull request to submit that fix. (EDIT: done in #4687)

codecov · 2018-04-27T18:59:41Z

Codecov Report

Merging #4685 into master will increase coverage by 0.05%.
The diff coverage is 98.14%.

@@            Coverage Diff             @@
##           master    #4685      +/-   ##
==========================================
+ Coverage    82.6%   82.65%   +0.05%     
==========================================
  Files         140      141       +1     
  Lines       15868    15922      +54     
  Branches     1983     1987       +4     
==========================================
+ Hits        13107    13160      +53     
  Misses       2567     2567              
- Partials      194      195       +1

Impacted Files	Coverage Δ
celery/app/backends.py	`59.37% <ø> (ø)`	⬆️
celery/app/defaults.py	`95.45% <ø> (ø)`	⬆️
celery/backends/azureblockblob.py	`98.14% <98.14%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c753d7...fdd661b. Read the comment docs.

georgepsarakis · 2018-04-28T05:32:59Z

@c-w thank you very much for your contribution! Is there any chance that we can run integration/functional test with this backend, without actually connecting to Azure?

c-w · 2018-04-28T17:31:00Z

Hi @georgepsarakis. There is Azurite for local emulation of Azure Storage which should be able to emulate all the APIs used in this pull request. If you wish, I can look into setting up Azurite for integration tests.

How often do you run the full integration test suite? Are you fundamentally opposed to making live calls against a backend or is it just an issue of setting up test credentials and the associated cost?

georgepsarakis · 2018-04-28T17:42:51Z

@c-w Azurite seems as a great choice for running the tests.

@thedrow @auvipy what are your thoughts on this? I believe this could be merged (probably in a later version than 4.2), once we have a closer look. I am just asking if you are agreed, in order to continue looking into the integration testing option. Thanks.

auvipy · 2018-04-28T18:06:32Z

as its a bit big feature we should skip 4.2

c-w · 2018-04-28T18:31:56Z

That's great to hear. I'll get onto the integration tests via Azurite and push them onto this pull request.

We were also planning to make a few follow-up pull requests after this one:

A CosmosDB storage backend.
A Storage Queue transport channel.
A ServiceBus transport channel.

The overall idea is to use the Storage Queue transport channel with the Block Blob storage backend for simple workloads since the services are very cheap and combined only require setting up a single Azure resource. The combination of ServiceBus transport channel and CosmosDB storage backend on the other hand would enable more demanding workloads due to offering stronger latency guarantees, multi-zone failover, etc.

We already have some branches with work-in-progress code for each of these features, but it'll take a bit more time to clean them up and submit pull requests. If you have any thoughts about the above, just let me know.

c-w · 2018-04-28T21:35:23Z

@georgepsarakis I've implemented the end-to-end test for the backend in 3ea28b6 and 1b9a263.

Travis CI now starts Azurite via Docker:

The required environment variables to light up the end-to-end test are then also set and the test executes:

Changing between Azurite and live Azure with this mechanism is also easy (just over-write the AZUREBLOCKBLOB_URL environment variable with an encrypted Travis environment variable).

auvipy · 2018-04-29T04:35:44Z

great work ! @c-w

georgepsarakis · 2018-04-29T04:40:36Z

Excellent work @c-w . I will have a closer look to the code and let you know if there are any suggestions.

georgepsarakis

@c-w some minor notes, generally this looks fine.

georgepsarakis · 2018-04-29T04:52:43Z

          fi
+    - |
+          docker run -d -e executable=blob -t -p 10000:10000 arafato/azurite:2.6.5
+          while ! nc -zv 127.0.0.1 10000; do sleep 10; done


Very simple and efficient way of testing when the listening port is available 👍 .

georgepsarakis · 2018-04-29T04:54:37Z

+    - |
+          docker run -d -e executable=blob -t -p 10000:10000 arafato/azurite:2.6.5
+          while ! nc -zv 127.0.0.1 10000; do sleep 10; done
+          export AZUREBLOCKBLOB_URL="azureblockblob://DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;"


I am guessing the AccountKey parameter is a random value. Does it have to be in a specific format or we could use something that is more clearly a testing sample value?

Unfortunately this is a required "well-known" value. See the key authentication section in the official docs for the storage emulator.

I see, great, thanks for clarifying.

georgepsarakis · 2018-04-29T05:07:59Z

+
+        self._container_name = (
+            container_name or
+            conf.get(CONF_CONTAINER_NAME) or


It's slightly simpler to just use the default value of the get method here:

conf.get(CONF_CONTAINER_NAME, DEFAULT_CONTAINER_NAME)

That was my initial implementation but it failed the tests since the conf value were explicitly set to None in one of the mappings that the conf access checks. Moving the default values to app/defaults.py fixed this. Implemented in 44517e3.

georgepsarakis · 2018-04-29T05:08:30Z

+
+        self._retry_initial_backoff_sec = (
+            retry_initial_backoff_sec or
+            conf.get(CONF_RETRY_INITIAL_BACKOFF_SEC) or


Same comment as above for all configuration options.

Implemented in 44517e3.

georgepsarakis · 2018-04-29T05:10:25Z

+            container_name=self._container_name, fail_on_exist=False)
+
+        if created:
+            LOGGER.info("Created container %s", self._container_name)


You can also specify that this action refers to Azure Block Blob storage in the log message.

Done in f91b69b.

georgepsarakis · 2018-04-29T05:35:55Z

@@ -0,0 +1,3 @@
+azure-storage>=0.36.0


It is not advisable to use non-exact tags for package versions, as they might lead to inconsistencies among installations and CI results. Could you please use the most recent usable version?

Could you provide some more guidance on what you require here? The azure-storage SDK has a bit of an odd versioning scheme where all versions below 0.36.0 are considered deprecated and the "new" SDK is versions 0.36.0 and above (see compatibility section on the official repository). As such, this dependency statement basically means "use any of the new SDK versions". Would you like to pin this to a specific version instead?

Exactly, my suggestion was to pin the version for the new dependencies to the exact latest:

azure-storage==0.36.0 azure-common==... azure-storage-common==...

Alright, done in ce12efd.

georgepsarakis · 2018-04-29T05:40:54Z

+
+
+@contextlib.contextmanager
+def patch_client():


I am not sure, but I don't think you need this context manager. patch will work as you require, see this example of mocking an instance method.

Great catch! Done in 16ce838.

georgepsarakis · 2018-04-29T05:44:47Z

+        with patch_client() as mock_client:
+            self.backend.get(b"mykey")
+
+            mock_client.get_blob_to_text.assert_has_calls(


I think that perhaps assert_called_once_with is more suitable here.

Done in 16ce838.

georgepsarakis · 2018-04-29T05:46:47Z

+            mock_client.delete_blob.assert_called_once_with(
+                "celery", "mykey")
+
+    @skip.unless_environ("AZUREBLOCKBLOB_URL")


If I understand correctly, this test case probably belongs to the integration test suite.

I'm having some issues running the integration tests locally via docker-compose so I've made a change and pushed it up to Travis in 9ba725d to see if the move of the test was successful. I'll tag you on the PR when everything is ready to be re-reviewed.

Alright, confirmed that integration tests ran and the new backend test executed and passed. Sample CI job log: https://travis-ci.org/celery/celery/jobs/372769687

georgepsarakis · 2018-04-29T05:52:26Z

    dynamodb: TEST_BACKEND=dynamodb://@localhost:8000
    dynamodb: AWS_ACCESS_KEY_ID=test_aws_key_id
    dynamodb: AWS_SECRET_ACCESS_KEY=test_aws_secret_key
-PASSENV =


The conversion to lowercase is an improvement of course, but as a general note, I believe it is preferable to avoid making changes outside the scope of the PR (however small), especially in extensive feature requests, as they might confuse reviewers and anyone inspecting the commit history in the future. Again, just explaining my view, no need to revert this 😄 !

Totally agree on keeping diffs minimal, I'm a stickler for that in all of my code reviews, so we're definitely aligned there ;-)

However, I do believe that this is a required change. I was investigating for a while why the live test would still be skipped despite the AZUREBLOCKBLOB_URL variable being set, and it ended up being linked to the environment isolation that tox uses. Simply adding the AZUREBLOCKBLOB_URL to the passenv section though didn't fix it since the passenv property apparently needs to be lowercased (which is surprising given that configparser is supposed to be case insensitive).

Wasn't aware of that, thanks for fixing this then 😄 !

c-w · 2018-04-30T09:23:24Z

Hi @georgepsarakis. I believe I've addressed all the comments. Could you please take another look?

georgepsarakis · 2018-04-30T18:51:08Z

 .. _`CouchDB`: http://www.couchdb.com/
 .. _`Couchbase`: https://www.couchbase.com/
 .. _`Consul`: https://consul.io/
+.. _`AzureBlockBlob`: https://azure.microsoft.com/en-us/services/storage/


Correct me if I am wrong, but I think I found a more accurate link here.

Done in e6284b0.

georgepsarakis · 2018-04-30T18:57:25Z

+
+from celery.backends.azureblockblob import AzureBlockBlobBackend
+
+


Although this test has certainly value, it is better if you also add and activate the integration suite by changing the tox configuration. You simply need to specify TEST_BROKER and TEST_BACKEND variables accordingly.

Done in 446130f. Will verify on Travis and ping you once it's ready to be reviewed.

There are quite a few backends that don't run a full integration test though (e.g. cassandra, elasticsearch or mongodb). Is it really worth extending the CI run substantially for this integration test when the simple backend test already gives us confidence that the backend works? If not: we might even have an opportunity here to speed up the CI suite by moving the DynamoDB test (and all other Key-Value Store backends) to the new backend integration test style.

I have a plan to use seaworthy for the integration tests so that we could start our services from within the test suite.
Please don't change how the test suite works yet.

The backend is implemented on top of the azure-storage library [1] which uses Azure Blob Storage [2] for a scalable low-cost PaaS backend. The backend was load tested via a simple nginx/gunicorn/sanic app hosted on a DS4 virtual machine (4 vCores, 16 GB RAM) and was able to handle 600+ concurrent users at ~170 RPS. The commit also contains a live end-to-end test to facilitate verification of the backend functionality. The test is activated by setting the `AZUREBLOCKBLOB_URL` environment variable to `azureblockblob://{ConnectionString}` where the value for `ConnectionString` can be found in the `Access Keys` pane of a Storage Account resources in the Azure Portal. [1] https://github.com/Azure/azure-storage-python [2] https://azure.microsoft.com/en-us/services/storage/

c-w · 2018-05-01T02:28:25Z

This should now be ready for the next round of reviews, @georgepsarakis.

georgepsarakis · 2018-05-01T05:59:19Z

@c-w thank you very much for the effort and changes so far. Integration tests seem to be running.
However builds seem a bit slow. Would you like to give a try to Docker container tmpfs option?

docker run -d -e executable=blob --tmpfs /opt --tmpfs /tmp --tmpfs /var -t -p 10000:10000 arafato/azurite:2.6.5

DynamoDB is started without persistence so I think there is not much to be done, however the same may apply:

docker run -d --tmpfs /tmp --tmpfs /opt -p 8000:8000 dwmkerr/dynamodb:38

What do you think?

c-w · 2018-05-01T07:29:03Z

That's very interesting. I didn't know about tmpfs in Docker; seems like a neat feature. I did a bit of research and used docker diff {container} to find the mutated directories of the dynamodb and azurite containers and mapped those to memory (so that we avoid potentially confusing extra declarations). Let's see if that speeds up the build time from the current ~8 minutes for azureblockblob and ~4 minutes for dynamodb.

Alternatively, as I mentioned in #4685 (comment), we could also consider turning off the full integration tests for azureblockblob and dynamodb and instead only rely on the new backend integration test which only takes seconds to run. The idea here would be to use the core redis and rabbitmq integration tests to verify that the framework wiring is working as expected and then to verify additional backends like azureblockblob, dynamodb, cassandra, etc. only at the backend interface layer (under the assumption that if the backend contract is correctly implemented, the rest of the framework wiring will work as expected). This should massively speed up the build at a small loss of end-to-end verification. Thoughts?

Another thought: would it make sense to add the .tox directory to the Travis cache?

c-w · 2018-05-01T07:58:33Z

Looks like we save about 60% execution time on the azureblockblob integration tests by using tmpfs, so that was a great suggestion!

Pre-tmpfs CI has ~8 minutes per azureblockblob task.
Post-tmpfs CI has ~3 minutes per azureblockblob task.

Overall the CI now takes still quite a bit longer though due to the new integration tests:

Pre-azureblockblob CI has ~20 minutes for the full suite wall time, ~75 minutes serial time.
Post-azureblockblob CI has ~30 minutes for the full suite wall time, ~100 minutes serial time.

Let me know if this is acceptable, @georgepsarakis, or if you'd prefer to try some more optimizations as described in #4685 (comment).

thedrow · 2018-05-01T08:27:15Z

@c-w The .tox directory is quite large and caching it might not provide us with the benefit we're after. In any case, if we choose to do so we'll need to also install and use tox-pip-extensions to avoid recreating the virtual environment every time requirements change.
I'd rather discuss this after we merge this PR.

At this point I'd rather extend our CI time than to loose the guarantee that everything works as expected.
We need to include every result backend we have and broker we support in the integration test suite.
Celery had quite a few bugs that were uncovered very late due to infrequent use of some of the result backends.
In Celery 5, we will split those result backends to different packages. They'll have a separate CI process which will help us reduce the build time.

c-w · 2018-05-01T08:59:27Z

@thedrow Excellent, thanks for the clarification. It would definitely be neat to split out the results backends into their own repo (just like how the brokers are currently split out into the kombu repository). So is there anything else you need me to do before you can consider this ready to merge?

thedrow

Overall this is very good work.
There are a few minor issues I'd like to address first.

thedrow · 2018-05-01T10:11:21Z

-PASSENV =
+
+    azureblockblob: TEST_BROKER=redis://
+    azureblockblob: TEST_BACKEND=azureblockblob://DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;


What's this account? Is it available to us forever?
If not, the secrets should not appear here.

This is a "well known" string to connect to the Azurite storage emulator, not a real secret. See earlier discussion #4685 (comment) or the official docs.

thedrow · 2018-05-01T10:13:43Z

+@skip.unless_module("azure")
+@skip.unless_environ("AZUREBLOCKBLOB_URL")
+class test_AzureBlockBlobBackend:
+    def test_live(self, manager):


Integration tests usually take a long time to complete.
This test checks two different cases. Let's split it to two tests.

Done in 0167476. NB: the test seems to take less than 1 second on Travis currently so the runtime is not too bad (I guess because it's hitting the backend directly instead of also going through a more complex workflow with the broker and the rest of the platform).

thedrow · 2018-05-01T10:13:55Z

+            app=manager.app,
+            url=os.environ["AZUREBLOCKBLOB_URL"])
+
+        key_values = dict((("akey%d" % i).encode(), "avalue%d" % i)


Please use a dict comprehension.

Done in 0167476.

thedrow · 2018-05-01T10:15:14Z

+
+        actual_values = backend.mget(key_values.keys())
+        for value, actual_value in zip(key_values.values(), actual_values):
+            assert value == actual_value


Please assert all(value == actual_value for value, actual_value in zip(key_values.values(), actual_values).

You can add a custom message to show the difference between both.

Done in 0167476. The custom message is formatted by py.test for array comparison already.

thedrow · 2018-05-01T10:17:20Z

+
+    @classmethod
+    def _parse_url(cls, url):
+        match = URL_RE.match(url)


We usually use urllib.parse as far as I can recall.
Why do we need a regex here?

Good catch. Refactored in 4dc8f31.

thedrow

LGTM.
Let's merge this after 4.2 hits GA.

georgepsarakis

Nice work @c-w , thank you very much for your contribution!

@ankurokok

* Add transports based on Azure PaaS This pull request adds two new transport implementations: - `azurestoragequeues` is implemented on top of Azure Storage Queues [1]. This offers a simple but scalable and low-cost PaaS transport for Celery users in Azure. The transport is intended to be used in conjunction with the Azure Block Blob Storage backend [2]. - `azureservicebus` is implemented on top of Azure Service Bus [3] and offers PaaS support for more demanding Celery workloads in Azure. The transport is intended to be used in conjunction with the Azure CosmosDB backend [4]. This pull request was created together with @ankurokok, @dkisselev, @evandropaula, @martinpeck and @michaelperel. [1] https://azure.microsoft.com/en-us/services/storage/queues/ [2] celery/celery#4685 [3] https://azure.microsoft.com/en-us/services/service-bus/ [4] celery/celery#4720 * Exclude Azure transports from code coverage There is test coverage for the transports but the tests require Azure credentials to run (passed via environment variables) so codecov doesn't exercise them. * Remove env vars to configure transport * Remove abbreviations

auvipy · 2018-08-30T17:44:39Z

@c-w thanks! and please don't forget to chime in when azure related issues arise!

This reverts commit dfc9b2e.

auvipy · 2018-08-30T17:57:20Z

should we revert this before kombu 4.3 release?

c-w · 2018-08-30T19:57:33Z

Sure thing, @auvipy; feel free to tag me on Azure-related issues.

auvipy · 2018-08-31T06:47:44Z

Clemens, could you investigate the reasons of build failure after merging this PR?

This reverts commit dfc9b2e.

thedrow · 2018-09-02T12:25:49Z

Was this reverted because of test failures?

auvipy · 2018-09-02T15:16:55Z

yes! I was confused, and reverted back!

thedrow · 2018-09-03T11:19:28Z

So we need a new PR with the changes in this PR right?

This reverts commit 8e6b2bf.

auvipy · 2018-09-03T11:39:08Z

yes!! what I did is again reverted the reverted pr! let's see what the build says!

c-w · 2018-09-03T16:41:27Z

Looking into it now, @auvipy.

c-w · 2018-09-03T17:22:48Z

Looks like the build failure is linked to Travis CI setup. When I originally opened this PR, Docker had to be manually installed via apt-get on Travis and since then this requirement was removed (which makes the apt-get code now fail). Removing the custom Docker setup on Travis makes the CI proceed as normal.

Here's the diff for the fix:

---
 .travis.yml | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 2e30d75..7760a6f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -98,9 +98,3 @@ services:
     - rabbitmq
     - redis
     - docker
-addons:
-  apt:
-    sources:
-      - debian-sid
-    packages:
-      - docker-ce
--
2.7.4

I'm now waiting for a full build to go through and will submit a follow-up PR once that's done.

auvipy · 2018-09-03T17:40:12Z

great thanks for investigating!

c-w · 2018-09-03T17:40:17Z

Here's an example of a Travis build with the Azure Block Blob backend change and the Travis CI fix mentioned above: https://travis-ci.org/CatalystCode/celery/builds/424006344. For some reason, the tests for the Redis backend on Python 3 are now failing which I find surprising. I'll investigate why this is the case.

c-w · 2018-09-03T18:58:33Z

Looks like the Redis test failures are transient and linked to timeouts/timing. Restarting the build fixes the test runs. Submitted the new pull request: #5025

* Revert "Revert "Add Azure Block Blob Storage backend (#4685)" (#5015)" This reverts commit 8e6b2bf. * Remove custom docker install on Travis Docker now has first class support on Travis so the custom apt-get install instructions are no longer required and make the build fail.

auvipy reviewed Apr 27, 2018

View reviewed changes

georgepsarakis suggested changes Apr 29, 2018

View reviewed changes

georgepsarakis suggested changes Apr 30, 2018

View reviewed changes

c-w added 3 commits May 1, 2018 00:42

Make Azure link more specific

e6284b0

Run full integration test with Azure Blob backend

446130f

Move dynamodb and azurite storage to memory

c1d15a5

thedrow suggested changes May 1, 2018

View reviewed changes

Refactor backend integration test

0167476

thedrow approved these changes May 1, 2018

View reviewed changes

georgepsarakis approved these changes May 1, 2018

View reviewed changes

Replace regex connection string parsing

fdd661b

c-w mentioned this pull request May 20, 2018

Add CosmosDB storage backend #4720

Closed

georgepsarakis added this to the v4.3 milestone May 25, 2018

c-w mentioned this pull request Jun 18, 2018

Add transports based on Azure PaaS celery/kombu#891

Merged

auvipy approved these changes Aug 30, 2018

View reviewed changes

auvipy merged commit dfc9b2e into celery:master Aug 30, 2018

auvipy added a commit that referenced this pull request Aug 30, 2018

Revert "Add Azure Block Blob Storage backend (#4685)"

56f654c

This reverts commit dfc9b2e.

auvipy mentioned this pull request Aug 30, 2018

Revert "Add Azure Block Blob Storage backend" #5015

Merged

c-w deleted the azureblockblob-backend branch August 30, 2018 19:58

auvipy added a commit that referenced this pull request Sep 1, 2018

Revert "Add Azure Block Blob Storage backend (#4685)" (#5015)

8e6b2bf

This reverts commit dfc9b2e.

auvipy added a commit that referenced this pull request Sep 3, 2018

Revert "Revert "Add Azure Block Blob Storage backend (#4685)" (#5015)"

0dd8273

This reverts commit 8e6b2bf.

c-w mentioned this pull request Sep 3, 2018

Un-revert Azure Block Blob backend #5025

Merged


		from celery.backends.azureblockblob import AzureBlockBlobBackend

Uh oh!

Uh oh!

Conversation

c-w commented Apr 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

auvipy left a comment

Choose a reason for hiding this comment

Uh oh!

c-w commented Apr 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Apr 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

georgepsarakis commented Apr 28, 2018

Uh oh!

c-w commented Apr 28, 2018

Uh oh!

georgepsarakis commented Apr 28, 2018

Uh oh!

auvipy commented Apr 28, 2018

Uh oh!

c-w commented Apr 28, 2018

Uh oh!

c-w commented Apr 28, 2018

Uh oh!

auvipy commented Apr 29, 2018

Uh oh!

georgepsarakis commented Apr 29, 2018

Uh oh!

georgepsarakis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

c-w commented Apr 27, 2018 •

edited

Loading

c-w commented Apr 27, 2018 •

edited

Loading

codecov Bot commented Apr 27, 2018 •

edited

Loading

c-w May 1, 2018 •

edited

Loading

c-w commented May 1, 2018 •

edited

Loading

c-w commented May 1, 2018 •

edited

Loading

c-w commented May 1, 2018 •

edited

Loading