Implement file-of-patterns functionality #444

tneymanov · 2019-03-05T21:55:35Z

Allows users to specify a list of files or input pattern inside a file to ingest into the tool.

Notes:

Could be helpful for cases when there isn't an easy regex for users to select all of their files.
VCF to BQ preprocessor is also modified to accept both inputs.
Using input_file flag method automatically quasi-turns on optimize_for_large_inputs, so it would be preferable not to use it for inputs of less than 50k files.

Issue: 441

coveralls · 2019-03-05T22:04:48Z

Pull Request Test Coverage Report for Build 1650

119 of 143 (83.22%) changed or added relevant lines in 8 files are covered.
12 unchanged lines in 5 files lost coverage.
Overall coverage increased (+0.09%) to 89.025%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
gcp_variant_transforms/testing/integration/run_preprocessor_tests.py	0	2	0.0%
gcp_variant_transforms/testing/integration/run_vcf_to_bq_tests.py	0	2	0.0%
gcp_variant_transforms/pipeline_common.py	24	30	80.0%
gcp_variant_transforms/vcf_to_bq_preprocess.py	0	6	0.0%
gcp_variant_transforms/vcf_to_bq.py	4	12	33.33%

Files with Coverage Reduction	New Missed Lines	%
gcp_variant_transforms/vcf_to_bq_preprocess.py	1	0.0%
gcp_variant_transforms/testing/integration/run_vcf_to_bq_tests.py	1	26.49%
gcp_variant_transforms/testing/integration/run_preprocessor_tests.py	1	27.85%
gcp_variant_transforms/pipeline_common.py	2	70.83%
gcp_variant_transforms/vcf_to_bq.py	7	36.07%

Totals
Change from base Build 1638:	0.09%
Covered Lines:	7130
Relevant Lines:	8009

💛 - Coveralls

allieychen

Thanks Tural for adding input file option and detailed unit tests! I have added some comments. Please also add the related issue (#441) in the description.

gcp_variant_transforms/options/variant_transform_options.py

...ng/integration/vcf_to_bq_tests/presubmit_tests/small_tests/combine_from_multiple_inputs.json

allieychen · 2019-03-06T14:49:48Z

...gration/vcf_to_bq_tests/presubmit_tests/small_tests/merge_option_move_to_call_from_file.json

@@ -0,0 +1,31 @@
+[


I would recommend to have the integration tests for the following cases:

vcf to bq

preprocessor

vcf to bq with annotation

The above cases would be affected by the input_file since it provides different sources for variants. On the other hand, merge_option should be fine as long as we can read the variants correctly.

Done, running the tests right now.

I don't see the point of adding this test. It won't catch more errors than existing test cases (i.e., the above test case and merge_option_info_keys_to_move_to_calls_regex would cover possible the same failure scenarios.)

gcp_variant_transforms/pipeline_common.py

tneymanov

Thanks for review, applied the changes. Running the new integration tests rn.

gcp_variant_transforms/options/variant_transform_options.py

tneymanov · 2019-03-07T17:16:47Z

...gration/vcf_to_bq_tests/presubmit_tests/small_tests/merge_option_move_to_call_from_file.json

@@ -0,0 +1,31 @@
+[


Done, running the tests right now.

...ng/integration/vcf_to_bq_tests/presubmit_tests/small_tests/combine_from_multiple_inputs.json

gcp_variant_transforms/pipeline_common.py

kemp-google · 2019-03-08T18:43:49Z

gcp_variant_transforms/options/variant_transform_options.py

 from gcp_variant_transforms.beam_io import vcfio
 from gcp_variant_transforms.libs import bigquery_sanitizer

-


I thought python style was to keep two blank lines?

kemp-google · 2019-03-08T18:44:22Z

gcp_variant_transforms/options/variant_transform_options.py

+    parser.add_argument(
+        '--input_file',
+        help=('File that contains the list of VCF file names to input. Either '
+              'this or --input_pattern flag has to be provided, exlusively.'


typo (exlusively)

gcp_variant_transforms/options/variant_transform_options.py

kemp-google · 2019-03-08T18:45:29Z

gcp_variant_transforms/options/variant_transform_options.py

+        '--input_file',
+        help=('File that contains the list of VCF file names to input. Either '
+              'this or --input_pattern flag has to be provided, exlusively.'
+              'Note that using input_file than input_pattern is slower for '


s/than/rather than/

gcp_variant_transforms/options/variant_transform_options.py

kemp-google · 2019-03-08T18:47:17Z

gcp_variant_transforms/options/variant_transform_options.py

+  if parsed_args.input_pattern is not None:
+    try:
+      # Gets at most 1 pattern match result of type `filesystems.MatchResult`.
+      first_match = filesystems.FileSystems.match(


If there are no matches, does the array access not fail?

gcp_variant_transforms/pipeline_common.py

kemp-google · 2019-03-08T18:50:35Z

gcp_variant_transforms/pipeline_common.py

-def read_headers(pipeline, pipeline_mode, input_pattern):
+def get_file_names(input_file):
+  # type (str) -> List(str)
+  """ Reads all input file and extracts list of patterns out of it."""


All this does it takes a file and returns a list of the lines, correct? Is there no python builtin for this? (something like open("foo").readlines() or so?)

kemp-google · 2019-03-08T18:51:15Z

gcp_variant_transforms/testing/integration/run_vcf_to_bq_tests.py

 To define a new integration test case, create a json file in
 `gcp_variant_transforms/testing/integration/vcf_to_bq_tests` directory and
-specify at least test_name, table_name, and input_pattern for the integration
+specify at least test_name and table_name for the integration


unrelated change?

Nope, related - this file was preventing us from running tests, since it assumed that input_pattern was a necessity. I've removed that requirement and updated the doc here to reflect that.

tneymanov

Due to commit override, github doesn't seem to allow me to reply to some of the Aaron's comments, so I'll do so here

I thought python style was to keep two blank lines?

Done.

All this does it takes a file and returns a list of the lines, correct? Is there no python builtin for this? (something like open("foo").readlines() or so?)

soo there is in built in python (splitlines()), but doesn't seem to be in apache beam's filesystems.

If there are no matches, does the array access not fail?

I haven't touched that part but for the each supplied input pattern, match returns a list of MatchResult object, each containing the pattern searched and the list of files it matched (metadata_list). The amount of items that are extracted in metadata_list can be upper-constrained to reduce run time, by passing list of limits to for each search (as we do here by passing [1] for single input pattern and [1,1,...,1] for input_file, since we only want to check existence here). if no matches are found for pattern, metadata_list is None.

Also verified with mock run.

I was actually hoping we could do something like this for the flags themselves, but that will only work if we get a chance to read the file early in the process

Yeah, as discussed offline, there is no obvious way to differentiate between directory and file in the Filesystems package. We can look deeper into it later.

Update: I also moved the input_patterns extraction logic into the argument parser step and created a new arg for it, to be used instead of input_pattern and input_file. I still use input_file field in get_pipeline_mode() step, but only to have more extensive error message - if it's not required, we can cut it out as well.

Integration tests passed.

gcp_variant_transforms/options/variant_transform_options.py

tneymanov · 2019-03-11T17:37:10Z

gcp_variant_transforms/options/variant_transform_options.py

+    parser.add_argument(
+        '--input_file',
+        help=('File that contains the list of VCF file names to input. Either '
+              'this or --input_pattern flag has to be provided, exlusively.'


tneymanov · 2019-03-11T17:37:32Z

gcp_variant_transforms/options/variant_transform_options.py

+        '--input_file',
+        help=('File that contains the list of VCF file names to input. Either '
+              'this or --input_pattern flag has to be provided, exlusively.'
+              'Note that using input_file than input_pattern is slower for '


gcp_variant_transforms/options/variant_transform_options.py

tneymanov · 2019-03-11T17:48:50Z

gcp_variant_transforms/testing/integration/run_vcf_to_bq_tests.py

 To define a new integration test case, create a json file in
 `gcp_variant_transforms/testing/integration/vcf_to_bq_tests` directory and
-specify at least test_name, table_name, and input_pattern for the integration
+specify at least test_name and table_name for the integration


Nope, related - this file was preventing us from running tests, since it assumed that input_pattern was a necessity. I've removed that requirement and updated the doc here to reflect that.

gcp_variant_transforms/pipeline_common.py

allieychen

Thanks Tural, I added a few comments :)

allieychen · 2019-03-12T20:15:39Z

gcp_variant_transforms/options/variant_transform_options.py

+      (not parsed_args.input_pattern and not parsed_args.input_file)):
+    raise ValueError('Exactly one of input_pattern and input_file has to be '
+                     'provided.')
+  if parsed_args.input_pattern is not None:


Can be simplified as if parsed_args.input_pattern:

Since we also have similar validations in _get_input_patterns and _get_file_names (we have to validate the input pattern listed in the input_file as well), I prefer to move all validations except the first statement to _get_file_names. And the code can be simplified:

If input_file is provided, check whether it exists and read the contents and assign to known_args.input_pattern.

Otherwise, convert known_args.input_pattern to [known_args.input_patterns].

Now we can validate every item in input_pattern.

gcp_variant_transforms/pipeline_common.py

allieychen · 2019-03-12T20:44:41Z

gcp_variant_transforms/testing/data/input_files/combine

@@ -0,0 +1 @@
+gs://gcp-variant-transforms-testfiles/small_tests/valid-4.0*.vcf


It might be worthy more to have two file_patterns for the testing purpose.

It does - there is * which encapsulates .../valid-4.0.vcf, .../valid-4.0-one-more-filed.vcf and .../valid-4.0-new.vcf.

Sorry, what I mean is that we provide two patterns in this file, e.g.,
gs://gcp-variant-transforms-testfiles/small_tests/valid-4.0*.vcf
gs://gcp-variant-transforms-testfiles/small_tests/valid-4.1*.vcf

The new flag essentially allows the user to provide more than one pattern, while in this test case the user can actually use input_pattern=gs://gcp-variant-transforms-testfiles/small_tests/valid-4.0*.vcf

allieychen · 2019-03-12T20:49:08Z

gcp_variant_transforms/pipeline_common.py

  # type: (beam.Pipeline, int, str) -> pvalue.PCollection
  """Creates an initial PCollection by reading the VCF file headers."""
-  if pipeline_mode == PipelineModes.LARGE or input_file:
+  if pipeline_mode == PipelineModes.LARGE or len(input_patterns) > 1:


It might be easier to just change the pipeline_mode to LARGE in get_pipeline_mode when we have more than one input_pattern. But please check whether we use the pipeline mode anywhere else rather than reading the variants.

Alas, annotation also looks up the pipeline mode, while it can have different input pattern (specifically, single input pattern of the shard directory).

The annotation pipeline also uses the pipeline_mode to read variants, I don't think it is different from other cases, right?

I'm sorry, I wasn't clear - at this step we potentially already have sharded the output so only have 1 input pattern, but whether input was large or not is still relevant

gcp_variant_transforms/testing/data/input_files/wrong

allieychen · 2019-03-14T13:46:29Z

gcp_variant_transforms/testing/data/input_files/sample

@@ -0,0 +1,3 @@
+./gcp_variant_transforms/testing/data/vcf/valid-4.0.vcf


It is better to include this in unit tests (i.e., create temporary files there). This folder is mainly used for integration tests.

Could you clarify what you mean here - do you want me to dynamically generate this and wrong files? Doesn't that seem like an overkill, or is there a precedence for this?

As per offline convo, Done.

allieychen · 2019-03-14T13:52:03Z

...gration/vcf_to_bq_tests/presubmit_tests/small_tests/merge_option_move_to_call_from_file.json

@@ -0,0 +1,31 @@
+[


I don't see the point of adding this test. It won't catch more errors than existing test cases (i.e., the above test case and merge_option_info_keys_to_move_to_calls_regex would cover possible the same failure scenarios.)

...n/vcf_to_bq_tests/presubmit_tests/small_tests/platinum_NA12878_hg38_10K_lines_from_file.json

gcp_variant_transforms/vcf_to_bq.py

- replace all (input_pattern, input_file) inputs to input_patterns - add "combine" input file - refactor v_t_options file - add integration tests for preprocessor and vcf->bq with annotations

- Remove validation from pipeline_common. - Adjust tests accordingly. - Change input_pattern checks to treat '' as absent input, to simplify the code.

tneymanov

Unfortunately had to comment to every comment one by one again :(

I don't see the point of adding this test. It won't catch more errors than existing test cases (i.e., the above test case and merge_option_info_keys_to_move_to_calls_regex would cover possible the same failure scenarios.)

Done

- Removed merge test duplicate for _from_file option. - Modified combine and gnomad tests. - Updated the gs storage for said tests. - Remove legacy debugging code. - Refactor validation to use pipeline_common for extracting file names - Create files for unit tests on the fly. - Set PipelineMode to Large, when multiple input patterns are available. - Move input pattern/file validation to pipeline_common "file reading" stage to remove duplication.

allieychen

Thank you Tural. I added more comments there. One thing I don't feel comfortable is now we have both known_args.input_pattern and known_args.input_patterns. This would be prone to bug. Can you try to come up with a better name? Or maybe just update input_pattern rather then use another variable when user uses input_file.

gcp_variant_transforms/pipeline_common.py

gcp_variant_transforms/options/variant_transform_options_test.py

gcp_variant_transforms/pipeline_common.py

gcp_variant_transforms/vcf_to_bq.py

gcp_variant_transforms/testing/integration/run_preprocessor_tests.py

tneymanov

Applied the requested changes. Renamed input_patterns to all_patterns.

gcp_variant_transforms/options/variant_transform_options_test.py

gcp_variant_transforms/pipeline_common.py

...n/vcf_to_bq_tests/presubmit_tests/small_tests/platinum_NA12878_hg38_10K_lines_from_file.json

gcp_variant_transforms/vcf_to_bq.py

allieychen

Thank you Tural! Please take a look at the comments I just added. Otherwise LGTM.

allieychen · 2019-03-19T20:26:12Z

...ests/presubmit_tests/medium_tests/gnomad_genomes_GRCh38_chrX_head2500_run_vep_from_file.json

@@ -1,8 +1,8 @@
 [
  {
    "test_name": "gnomad-genomes-grch37-chr-x-head2500-run-vep",


This test has the same test_name and table_name as gnomad_genomes_GRCh37_chrX_head2500_run_vep.json, it will fail since we can not have the same dataflow job name, nor same table name.

allieychen · 2019-03-20T16:51:35Z

gcp_variant_transforms/options/variant_transform_options.py

              'is not recommended.'.format(vcfio.VcfParserType.PYVCF.name,
                                           vcfio.VcfParserType.NUCLEUS.name)))

+


nit: Please remove the extra blank line. Generally, two blank lines between top-level definitions, class definitions. One blank line between method definitions.

gcp_variant_transforms/options/variant_transform_options_test.py

allieychen · 2019-03-20T16:56:32Z

gcp_variant_transforms/pipeline_common.py

+
+
+def _get_file_names(input_file):
+  # type (str) -> List[str]


allieychen · 2019-03-20T17:00:20Z

gcp_variant_transforms/testing/integration/preprocessor_tests/no_conflicts_from_file.json

@@ -0,0 +1,14 @@
+[
+  {
+    "test_name": "no-conflicts",


Please use a unique test_name.

Since this test actually passes even when we wrongly provide input_pattern, so it is preferred to modify the input file to actually generate some conflicts information. For instance, you can add gs://gcp-variant-transforms-testfiles/small_tests/invalid-4.0-POS-empty.vcf into combine, the expected contents should be something similar to report_malformed_records.json

allieychen · 2019-03-20T17:05:53Z

gcp_variant_transforms/vcf_to_bq.py


-def _read_variants(input_pattern, pipeline, known_args, pipeline_mode):
+def _read_variants(all_patterns, pipeline, known_args, pipeline_mode):
  # type: (str, beam.Pipeline, argparse.Namespace, int) -> pvalue.PCollection


s/str/List[str]

allieychen · 2019-03-20T17:06:20Z

gcp_variant_transforms/vcf_to_bq.py



-def _add_inferred_headers(input_pattern,  # type: str
+def _add_inferred_headers(all_patterns,  # type: str


s/str/List[str]

allieychen · 2019-03-20T17:07:12Z

gcp_variant_transforms/vcf_to_bq.py


-def _annotate_vcf_files(input_pattern, known_args, pipeline_args):
+def _annotate_vcf_files(all_patterns, known_args, pipeline_args):
  # type: (str, argparse.Namespace, List[str]) -> str


s/str/List[str]

allieychen · 2019-03-20T17:07:50Z

gcp_variant_transforms/vcf_to_bq.py

-                   annotated_vcf_pattern=None):
+def _merge_headers(known_args, pipeline_args,
+                   pipeline_mode, annotated_vcf_pattern=None):
  # type: (str, argparse.Namespace, List[str], int, str) -> None


Please update the type.

tneymanov

Applied the requested changes.

tneymanov · 2019-03-20T17:47:16Z

...ests/presubmit_tests/medium_tests/gnomad_genomes_GRCh38_chrX_head2500_run_vep_from_file.json

@@ -1,8 +1,8 @@
 [
  {
    "test_name": "gnomad-genomes-grch37-chr-x-head2500-run-vep",


tneymanov · 2019-03-20T17:48:05Z

gcp_variant_transforms/pipeline_common.py

+
+
+def _get_file_names(input_file):
+  # type (str) -> List[str]


tneymanov · 2019-03-20T17:49:13Z

gcp_variant_transforms/vcf_to_bq.py


-def _read_variants(input_pattern, pipeline, known_args, pipeline_mode):
+def _read_variants(all_patterns, pipeline, known_args, pipeline_mode):
  # type: (str, beam.Pipeline, argparse.Namespace, int) -> pvalue.PCollection


tneymanov · 2019-03-20T17:58:32Z

gcp_variant_transforms/vcf_to_bq.py


-def _annotate_vcf_files(input_pattern, known_args, pipeline_args):
+def _annotate_vcf_files(all_patterns, known_args, pipeline_args):
  # type: (str, argparse.Namespace, List[str]) -> str


tneymanov · 2019-03-20T18:22:45Z

gcp_variant_transforms/testing/integration/preprocessor_tests/no_conflicts_from_file.json

@@ -0,0 +1,14 @@
+[
+  {
+    "test_name": "no-conflicts",


tneymanov · 2019-03-20T18:23:08Z

gcp_variant_transforms/vcf_to_bq.py



-def _add_inferred_headers(input_pattern,  # type: str
+def _add_inferred_headers(all_patterns,  # type: str


- type definitions - preprocessor integration test

kemp-google · 2019-03-21T16:20:16Z

gcp_variant_transforms/pipeline_common.py

               | vcf_header_io.ReadAllVcfHeaders())
  else:
-    headers = pipeline | vcf_header_io.ReadVcfHeaders(input_pattern)
+    headers = pipeline | vcf_header_io.ReadVcfHeaders(all_patterns[0])


Is this safe? What if all_patterns is empty? Are you just counting on this not being called unless there is at least one pattern in the list?

It should be - we actually had this discussion before hand with regards to having validation on multiple places or only at the beginning and decided that it probably isn't worth to copy an unreachable code all over the pipeline, so long as the validation in the beginning is done right. So yeah, I'm counting that all_patterns has been verified until this point and every pattern in all patterns has at least 1 match in the filesystem

tneymanov

Added integration tests for annotation pipeline, with and without sharding, which took quite a bit of trial and error, but eventually passed.

Running all the rest of integration tests now.

tneymanov · 2019-03-22T15:24:35Z

gcp_variant_transforms/pipeline_common.py

               | vcf_header_io.ReadAllVcfHeaders())
  else:
-    headers = pipeline | vcf_header_io.ReadVcfHeaders(input_pattern)
+    headers = pipeline | vcf_header_io.ReadVcfHeaders(all_patterns[0])


It should be - we actually had this discussion before hand with regards to having validation on multiple places or only at the beginning and decided that it probably isn't worth to copy an unreachable code all over the pipeline, so long as the validation in the beginning is done right. So yeah, I'm counting that all_patterns has been verified until this point and every pattern in all patterns has at least 1 match in the filesystem

allieychen

Thank you Tural. LGTM!

tneymanov assigned allieychen Mar 5, 2019

tneymanov requested review from allieychen and kemp-google March 5, 2019 21:55

allieychen reviewed Mar 6, 2019

View reviewed changes

tneymanov force-pushed the file_of_files branch from e7472ce to d7b5a1a Compare March 7, 2019 17:13

tneymanov commented Mar 7, 2019

View reviewed changes

tneymanov force-pushed the file_of_files branch 2 times, most recently from 7456524 to 003beec Compare March 8, 2019 02:59

kemp-google reviewed Mar 8, 2019

View reviewed changes

tneymanov force-pushed the file_of_files branch 5 times, most recently from 2ea46b8 to 3f2bfb5 Compare March 11, 2019 20:44

tneymanov commented Mar 11, 2019

View reviewed changes

allieychen reviewed Mar 12, 2019

View reviewed changes

tneymanov force-pushed the file_of_files branch 3 times, most recently from 260caa6 to fd5844e Compare March 13, 2019 20:19

allieychen reviewed Mar 14, 2019

View reviewed changes

gcp_variant_transforms/testing/data/input_files/wrong Outdated Show resolved Hide resolved

allieychen reviewed Mar 14, 2019

View reviewed changes

tneymanov force-pushed the file_of_files branch from fd5844e to f368a6d Compare March 14, 2019 16:10

tneymanov added 3 commits March 14, 2019 16:10

Implement file-of-patterns functionality.git

23bb4f0

Apply the requested changes to the PR.

7e4aeca

- replace all (input_pattern, input_file) inputs to input_patterns - add "combine" input file - refactor v_t_options file - add integration tests for preprocessor and vcf->bq with annotations

Address comments:

9a07239

- Remove validation from pipeline_common. - Adjust tests accordingly. - Change input_pattern checks to treat '' as absent input, to simplify the code.

tneymanov force-pushed the file_of_files branch 3 times, most recently from b44773d to 3223f34 Compare March 15, 2019 16:47

tneymanov commented Mar 15, 2019

View reviewed changes

tneymanov force-pushed the file_of_files branch from 3223f34 to ceb5709 Compare March 18, 2019 17:03

tneymanov force-pushed the file_of_files branch from ceb5709 to bf5f472 Compare March 18, 2019 18:23

allieychen reviewed Mar 19, 2019

View reviewed changes

Applied the requested changes.

dcee9a8

tneymanov commented Mar 19, 2019

View reviewed changes

allieychen reviewed Mar 20, 2019

View reviewed changes

tneymanov commented Mar 20, 2019

View reviewed changes

tneymanov force-pushed the file_of_files branch from 7040da7 to 4685940 Compare March 20, 2019 18:24

Applied the requested changes.

83d6ccf

- type definitions - preprocessor integration test

tneymanov force-pushed the file_of_files branch from 4685940 to 83d6ccf Compare March 20, 2019 20:16

kemp-google reviewed Mar 21, 2019

View reviewed changes

Add integration tests for annotation pipeline with and without sharding.

f01ff8a

tneymanov commented Mar 22, 2019

View reviewed changes

allieychen approved these changes Mar 22, 2019

View reviewed changes

tneymanov merged commit b0343a3 into googlegenomics:master Mar 25, 2019

		from gcp_variant_transforms.beam_io import vcfio
		from gcp_variant_transforms.libs import bigquery_sanitizer

		@@ -0,0 +1 @@
		gs://gcp-variant-transforms-testfiles/small_tests/valid-4.0*.vcf

		@@ -0,0 +1,3 @@
		./gcp_variant_transforms/testing/data/vcf/valid-4.0.vcf

		'is not recommended.'.format(vcfio.VcfParserType.PYVCF.name,
		vcfio.VcfParserType.NUCLEUS.name)))



		def _add_inferred_headers(input_pattern, # type: str
		def _add_inferred_headers(all_patterns, # type: str

Implement file-of-patterns functionality #444

Implement file-of-patterns functionality #444

Uh oh!

Conversation

tneymanov commented Mar 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Mar 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 1650

💛 - Coveralls

Uh oh!

allieychen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tneymanov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tneymanov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

allieychen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

tneymanov commented Mar 5, 2019 •

edited

Loading

coveralls commented Mar 5, 2019 •

edited

Loading