[BEAM-12767] Improve PipelineOption parsing UX #15338

steveniemitz · 2021-08-17T12:55:47Z

This adds two improvements to the PipelineOptionFactory command line parser:

Failure to parse complex types due to missing quotes is retried with the input automatically quoted
@JsonDeserialize is now supported on getters, the specified deserializer in the annotation will be used to parse the input string.

R: @lukecwik @kennknowles since you two are the last to make any changes here.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Twister2
Go	---	---
Java
Python	---	---
XLang		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java

codecov · 2021-08-17T18:57:40Z

Codecov Report

Merging #15338 (cbb363f) into master (12d624b) will increase coverage by 0.03%.
The diff coverage is n/a.

❗ Current head cbb363f differs from pull request most recent head 16224d3. Consider uploading reports for the commit 16224d3 to get more accurate results

@@            Coverage Diff             @@
##           master   #15338      +/-   ##
==========================================
+ Coverage   83.79%   83.83%   +0.03%     
==========================================
  Files         440      440              
  Lines       59804    59886      +82     
==========================================
+ Hits        50115    50207      +92     
+ Misses       9689     9679      -10

Impacted Files	Coverage Δ
...n/apache_beam/ml/gcp/recommendations_ai_test_it.py	`56.81% <0.00%> (-12.95%)`	⬇️
.../apache_beam/io/gcp/datastore/v1new/datastoreio.py	`86.45% <0.00%> (-0.99%)`	⬇️
sdks/python/apache_beam/io/gcp/bigquery.py	`75.53% <0.00%> (-0.85%)`	⬇️
sdks/python/apache_beam/io/gcp/bigquery_tools.py	`86.67% <0.00%> (-0.74%)`	⬇️
sdks/python/apache_beam/dataframe/frames.py	`94.87% <0.00%> (-0.07%)`	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	`88.95% <0.00%> (-0.06%)`	⬇️
setup.py	`0.00% <0.00%> (ø)`
...dks/python/apache_beam/options/pipeline_options.py	`95.16% <0.00%> (ø)`
.../python/apache_beam/io/gcp/resource_identifiers.py	`100.00% <0.00%> (ø)`
.../python/apache_beam/portability/api/metrics_pb2.py	`100.00% <0.00%> (ø)`
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80da0ec...16224d3. Read the comment docs.

steveniemitz · 2021-08-18T16:24:47Z

hm, this says there's changes requested still but I can't actually find them in the review... @lukecwik I think I addressed all your comments.

steveniemitz · 2021-08-18T18:32:02Z

Run Java PreCommit

lukecwik · 2021-08-18T20:29:46Z

hm, this says there's changes requested still but I can't actually find them in the review... @lukecwik I think I addressed all your comments.

Thats just what the UI will say forever until another review happens by the person who requested changes.

lukecwik

Is this ready to review yet since I don't see any logic around @JsonSerialize/@JsonIgnore as we discussed. Everything else looks pretty good.

lukecwik · 2021-08-18T20:33:07Z

sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java

+   *
+   * <p>If the getter method is annotated with {@link
+   * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified deserializer will be
+   * used, otherwise the default ObjectMapper deserialization strategy.


Suggested change

* used, otherwise the default ObjectMapper deserialization strategy.

* used, otherwise the default ObjectMapper deserialization strategy is used.

lukecwik · 2021-08-18T20:37:15Z

sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsFactoryTest.java

+        PipelineOptionsFactory.fromArgs("--complexType=test").as(OptionsWithJsonDeserialize1.class);
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(


ExpectedException is deprecated, please use assertThrows(..., lambda that throws exception);

lukecwik · 2021-08-18T20:38:59Z

sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java

@@ -1572,6 +1601,80 @@ public int compare(Method o1, Method o2) {
    return builder.build();
  }

+  private static JsonDeserializer<Object> deserializerForMethod(Method method, JavaType type)


nit: method comment

Didn't mean to LGTM

lukecwik

See my earlier review #15338 (review)

steveniemitz · 2021-08-18T21:02:26Z

I don't see any logic around @JsonSerialize/@JsonIgnore as we discussed.

I'm not sure what I'd be testing/adding logic for here? The places that serialize PipelineOptions I've seen pass the typed PipelineOptions object to an ObjectMapper directly, is there some other path you were thinking about?

never mind I see what you mean now, I didn't realize the PipelineOptions had a custom serializer/deserializer as well. Will update that.

lukecwik · 2021-08-19T18:09:22Z

I don't see any logic around @JsonSerialize/@JsonIgnore as we discussed.

I'm not sure what I'd be testing/adding logic for here? The places that serialize PipelineOptions I've seen pass the typed PipelineOptions object to an ObjectMapper directly, is there some other path you were thinking about?

never mind I see what you mean now, I didn't realize the PipelineOptions had a custom serializer/deserializer as well. Will update that.

Great, thanks.

steveniemitz · 2021-08-19T19:53:05Z

alright, all hooked up and I added some more tests to ProxyInvocationHandlerTest as well to test the round-trip.

steveniemitz · 2021-08-19T22:56:18Z

Run Java PreCommit

steveniemitz · 2021-08-19T22:57:16Z

Run Java_Examples_Dataflow PreCommit

steveniemitz · 2021-08-19T23:20:43Z

~~I'm still tracking down one more issue w/ serialization now when serializing custom ValueProviders~~

edit: nm there wasn't an issue, I was just using an out-of-date build on my tests =/

steveniemitz · 2021-08-20T04:28:23Z

Run Java_Examples_Dataflow PreCommit

steveniemitz · 2021-08-20T14:35:06Z

Run Java PreCommit

steveniemitz · 2021-08-20T16:25:14Z

@lukecwik should be good to review now.

lukecwik · 2021-08-30T21:04:51Z

Run Java PreCommit

steveniemitz · 2021-08-30T23:44:47Z

Run Java PreCommit

steveniemitz · 2021-08-30T23:45:49Z

Run Java_Examples_Dataflow PreCommit

steveniemitz · 2021-08-31T00:38:21Z

Run Java_Examples_Dataflow PreCommit

steveniemitz · 2021-08-31T00:51:20Z

Run Java PreCommit

steveniemitz · 2021-08-31T00:51:52Z

Run Java_Examples_Dataflow_Java11 PreCommit

steveniemitz · 2021-08-31T14:14:38Z

lol one day these tests will succeed...

steveniemitz · 2021-08-31T14:14:41Z

Run Java PreCommit

steveniemitz · 2021-08-31T14:15:03Z

Run Java_Examples_Dataflow PreCommit

steveniemitz · 2021-08-31T16:37:25Z

Run Java PreCommit

steveniemitz · 2021-08-31T18:00:49Z

@lukecwik finally jammed all the tests through!

lukecwik · 2021-08-31T18:46:47Z

Thanks, if you have some time and can look at why the test is flaking it would be greatly appreciated.

kw2542 · 2021-09-01T17:17:15Z

@steveniemitz @lukecwik This commit seems to be causing bunch of post commit tests to failure:

https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/172/
https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/1347/
https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/5024/

Should we revert this PR for now?

steveniemitz · 2021-09-01T17:19:03Z

hm, weird. Yeah feel free to revert, I can look into the failures

kw2542 · 2021-09-01T17:26:22Z

I am not 100% sure if this is the RC but looks like it.

java.lang.RuntimeException: Failed to invoke samza job
	at org.apache.beam.runners.samza.SamzaPipelineRunner.run(SamzaPipelineRunner.java:83)
	at org.apache.beam.runners.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:86)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Failed to serialize PipelineOptions
	at org.apache.beam.runners.samza.translation.ConfigBuilder.build(ConfigBuilder.java:110)
	at org.apache.beam.runners.samza.SamzaRunner.runPortablePipeline(SamzaRunner.java:93)
	at org.apache.beam.runners.samza.SamzaPipelineRunner.run(SamzaPipelineRunner.java:73)
	... 7 more
Caused by: java.lang.IllegalArgumentException: Failed to serialize PipelineOptions
	at org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:70)
	at org.apache.beam.runners.core.construction.SerializablePipelineOptions.<init>(SerializablePipelineOptions.java:44)
	at org.apache.beam.runners.samza.translation.ConfigBuilder.build(ConfigBuilder.java:102)
	... 9 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Error while populating display data for component 'org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData': null
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._wrapAsIOE(DefaultSerializerProvider.java:509)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:482)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
	at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4487)
	at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3742)
	at org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:68)
	... 11 more
Caused by: org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder$PopulateDisplayDataException: Error while populating display data for component 'org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData': null
	at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:784)
	at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.delegate(DisplayData.java:740)
	at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:157)
	at com.sun.proxy.$Proxy0.populateDisplayData(Unknown Source)
	at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:775)
	at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.access$100(DisplayData.java:703)
	at org.apache.beam.sdk.transforms.display.DisplayData.from(DisplayData.java:81)
	at org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:706)
	at org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:654)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
	... 15 more
Caused by: java.lang.NullPointerException
	at com.fasterxml.jackson.databind.DeserializationContext.isEnabled(DeserializationContext.java:395)
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer._checkFromStringCoercion(StdDeserializer.java:1364)
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer._parseIntPrimitive(StdDeserializer.java:728)
	at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$IntegerDeserializer.deserialize(NumberDeserializers.java:529)
	at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$IntegerDeserializer.deserialize(NumberDeserializers.java:506)
	at org.apache.beam.sdk.options.PipelineOptionsFactory.deserializeNode(PipelineOptionsFactory.java:1791)
	at org.apache.beam.sdk.options.ProxyInvocationHandler.getValueFromJson(ProxyInvocationHandler.java:517)
	at org.apache.beam.sdk.options.ProxyInvocationHandler.getValueFromJson(ProxyInvocationHandler.java:512)
	at org.apache.beam.sdk.options.ProxyInvocationHandler.access$400(ProxyInvocationHandler.java:93)
	at org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData.populateDisplayData(ProxyInvocationHandler.java:359)
	at org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:775)
	... 24 more
ERROR:root:java.lang.NullPointerException

This is the stacktrace in Samza runner that failed these tests, where a similar stacktrace in Flink and Sparker Runner, what do you think?

steveniemitz · 2021-09-01T17:35:20Z

I think the patch is actually really simple, I can get another PR going right now, or we can revert and re-apply with it fixed. I'm fine with either.

diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
index 030c56ba59..e20a8641b6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
@@ -44,6 +44,7 @@ import com.fasterxml.jackson.databind.node.TreeTraversingParser;
 import com.fasterxml.jackson.databind.ser.DefaultSerializerProvider;
 import com.fasterxml.jackson.databind.type.TypeBindings;
 import com.fasterxml.jackson.databind.util.SimpleBeanPropertyDefinition;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
 import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
 import java.beans.BeanInfo;
 import java.beans.IntrospectionException;
@@ -499,7 +500,8 @@ public class PipelineOptionsFactory {
 
   private static final DefaultDeserializationContext DESERIALIZATION_CONTEXT =
       new DefaultDeserializationContext.Impl(MAPPER.getDeserializationContext().getFactory())
-          .createInstance(MAPPER.getDeserializationConfig(), null, null);
+          .createInstance(
+              MAPPER.getDeserializationConfig(), new TokenBuffer(MAPPER, false).asParser(), null);

lukecwik · 2021-09-01T18:02:15Z

I think the patch is actually really simple, I can get another PR going right now, or we can revert and re-apply with it fixed. I'm fine with either.

diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
index 030c56ba59..e20a8641b6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
@@ -44,6 +44,7 @@ import com.fasterxml.jackson.databind.node.TreeTraversingParser;
 import com.fasterxml.jackson.databind.ser.DefaultSerializerProvider;
 import com.fasterxml.jackson.databind.type.TypeBindings;
 import com.fasterxml.jackson.databind.util.SimpleBeanPropertyDefinition;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
 import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
 import java.beans.BeanInfo;
 import java.beans.IntrospectionException;
@@ -499,7 +500,8 @@ public class PipelineOptionsFactory {
 
   private static final DefaultDeserializationContext DESERIALIZATION_CONTEXT =
       new DefaultDeserializationContext.Impl(MAPPER.getDeserializationContext().getFactory())
-          .createInstance(MAPPER.getDeserializationConfig(), null, null);
+          .createInstance(
+              MAPPER.getDeserializationConfig(), new TokenBuffer(MAPPER, false).asParser(), null);

You can get both PRs out (forward fix and rollback) and run the additional failing tests in the forward fix.

steveniemitz · 2021-09-01T18:06:03Z

and run the additional failing tests in the forward fix.

Forgive the noob question, is there a way to run these post-commit tests on a PR?

kw2542 · 2021-09-01T18:07:05Z

and run the additional failing tests in the forward fix.

Forgive the noob question, is there a way to run these post-commit tests on a PR?

I am testing out the rollback at #15438 where you can see how these tests are being triggered.

steveniemitz · 2021-09-01T18:10:15Z

Thanks!

steveniemitz · 2021-09-01T18:11:49Z

roll-forward PR up as well: #15439 , I'm not sure if the trigger phrase worked for that one though, maybe I don't have permissions?

I'd love to figure out how to reproduce this in a unit test as well, it seems like we're missing coverage over something pretty fundamental.

steveniemitz · 2021-09-01T18:52:35Z

looks like the failing tests are fixed in #15439 now, so we can either roll forward with that or roll back w/ the revert. I'd prefer to roll forward but I'll leave that up to you :)

kw2542 · 2021-09-01T19:10:10Z

looks like the failing tests are fixed in #15439 now, so we can either roll forward with that or roll back w/ the revert. I'd prefer to roll forward but I'll leave that up to you :)

+1, let's fix forward.

tvalentyn · 2021-09-01T19:10:43Z

Forgive the noob question, is there a way to run these post-commit tests on a PR?

There is a link in PR template: See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

steveniemitz commented Aug 17, 2021

View reviewed changes

sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java Outdated Show resolved Hide resolved

lukecwik requested changes Aug 17, 2021

View reviewed changes

sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java Show resolved Hide resolved

sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java Outdated Show resolved Hide resolved

steveniemitz force-pushed the better-option-parsing branch 2 times, most recently from 183b6d8 to 1c80016 Compare August 18, 2021 16:23

steveniemitz force-pushed the better-option-parsing branch from 1c80016 to 62d2909 Compare August 18, 2021 17:02

lukecwik reviewed Aug 18, 2021

View reviewed changes

lukecwik previously approved these changes Aug 18, 2021

View reviewed changes

lukecwik self-requested a review August 18, 2021 20:45

lukecwik requested changes Aug 18, 2021

View reviewed changes

steveniemitz force-pushed the better-option-parsing branch 2 times, most recently from 13807ab to c45ce17 Compare August 19, 2021 17:45

steveniemitz force-pushed the better-option-parsing branch 2 times, most recently from 7afc8c6 to 9e0c248 Compare August 19, 2021 19:21

steveniemitz force-pushed the better-option-parsing branch from 9e0c248 to 554b1ac Compare August 19, 2021 20:58

steveniemitz force-pushed the better-option-parsing branch from 554b1ac to bd17b09 Compare August 19, 2021 22:58

steveniemitz force-pushed the better-option-parsing branch from bd17b09 to 87f2c72 Compare August 20, 2021 15:37

steveniemitz force-pushed the better-option-parsing branch from f374523 to f9b9959 Compare August 31, 2021 00:09

[BEAM-12767] Improve PipelineOption parsing UX

16224d3

steveniemitz force-pushed the better-option-parsing branch from f9b9959 to 16224d3 Compare August 31, 2021 14:42

lukecwik merged commit 0592083 into apache:master Aug 31, 2021

steveniemitz mentioned this pull request Sep 1, 2021

[BEAM-12767] Add another unit test for PipelineOptions deserialization #15442

Merged

4 tasks

steveniemitz deleted the better-option-parsing branch September 1, 2021 19:36

	* used, otherwise the default ObjectMapper deserialization strategy.
	* used, otherwise the default ObjectMapper deserialization strategy is used.

[BEAM-12767] Improve PipelineOption parsing UX #15338

[BEAM-12767] Improve PipelineOption parsing UX #15338

Conversation

steveniemitz commented Aug 17, 2021

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

codecov bot commented Aug 17, 2021 • edited

Codecov Report

steveniemitz commented Aug 18, 2021

steveniemitz commented Aug 18, 2021

lukecwik commented Aug 18, 2021

lukecwik left a comment

Choose a reason for hiding this comment

lukecwik Aug 18, 2021

Choose a reason for hiding this comment

lukecwik Aug 18, 2021

Choose a reason for hiding this comment

lukecwik Aug 18, 2021

Choose a reason for hiding this comment

lukecwik left a comment

Choose a reason for hiding this comment

steveniemitz commented Aug 18, 2021 • edited

lukecwik commented Aug 19, 2021

steveniemitz commented Aug 19, 2021

steveniemitz commented Aug 19, 2021

steveniemitz commented Aug 19, 2021

steveniemitz commented Aug 19, 2021 • edited

steveniemitz commented Aug 20, 2021

steveniemitz commented Aug 20, 2021

steveniemitz commented Aug 20, 2021

lukecwik commented Aug 30, 2021

steveniemitz commented Aug 30, 2021

steveniemitz commented Aug 30, 2021

steveniemitz commented Aug 31, 2021

steveniemitz commented Aug 31, 2021

steveniemitz commented Aug 31, 2021

steveniemitz commented Aug 31, 2021

steveniemitz commented Aug 31, 2021

steveniemitz commented Aug 31, 2021

steveniemitz commented Aug 31, 2021

steveniemitz commented Aug 31, 2021

lukecwik commented Aug 31, 2021

kw2542 commented Sep 1, 2021 • edited

steveniemitz commented Sep 1, 2021

kw2542 commented Sep 1, 2021 • edited

steveniemitz commented Sep 1, 2021

lukecwik commented Sep 1, 2021

steveniemitz commented Sep 1, 2021

kw2542 commented Sep 1, 2021

steveniemitz commented Sep 1, 2021

steveniemitz commented Sep 1, 2021 • edited

steveniemitz commented Sep 1, 2021

kw2542 commented Sep 1, 2021

tvalentyn commented Sep 1, 2021

`ValidatesRunner` compliance status (on master branch)

codecov bot commented Aug 17, 2021 •

edited

steveniemitz commented Aug 18, 2021 •

edited

steveniemitz commented Aug 19, 2021 •

edited

kw2542 commented Sep 1, 2021 •

edited

kw2542 commented Sep 1, 2021 •

edited

steveniemitz commented Sep 1, 2021 •

edited