STM supports scala.js #1338

t5kaji · 2019-08-03T06:05:29Z

Issue : #803

now scala.js code with STM like the following works

  def run(arg: List[String]) =
    (for {
      intVar <- TRef.make(14)
      _ <- intVar.set(42)
      v <- intVar.get
    } yield v ).commit

Changes in code

Implement LockedRef class
- simple data structure to replace sychronized and java.util.concurrent.Semaphore for locking, which are not supported in scala.js
Replace forEach method calls with iterator-based implementation
- this is necessary since scala.js does not support jdk8 higher-order function for HashMap as of now
Replace synchronized and java.util.concurrent.Semaphore with LockedRef

Changes in performance

Comparison of benchmark results between master branch and this feature branch
- done by executing sbt:zio> benchmarks/jmh:run .*SemaphoreBenchmark.* task locally

master branch

[info] Result "zio.stm.SemaphoreBenchmark.semaphoreCatsContention":
[info]   141.237 ±(99.9%) 7.604 ops/s [Average]
[info]   (min, avg, max) = (120.844, 141.237, 155.422), stdev = 10.151
[info]   CI (99.9%): [133.633, 148.841] (assumes normal distribution)

[info] Result "zio.stm.SemaphoreBenchmark.semaphoreContention":
[info]   101.732 ±(99.9%) 5.197 ops/s [Average]
[info]   (min, avg, max) = (90.690, 101.732, 114.566), stdev = 6.938
[info]   CI (99.9%): [96.535, 106.929] (assumes normal distribution)

[info] Result "zio.stm.SemaphoreBenchmark.tsemaphoreContention":
[info]   114.949 ±(99.9%) 5.291 ops/s [Average]
[info]   (min, avg, max) = (98.940, 114.949, 124.836), stdev = 7.064
[info]   CI (99.9%): [109.658, 120.241] (assumes normal distribution)

[info] # Run complete. Total time: 00:25:06
[info] REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
[info] why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
[info] experiments, perform baseline and negative tests that provide experimental control, make sure
[info] the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
[info] Do not assume the numbers tell you what you want them to tell.
[info] Benchmark                                   (fibers)  (ops)   Mode  Cnt    Score   Error  Units
[info] SemaphoreBenchmark.semaphoreCatsContention        10   1000  thrpt   25  141.237 ± 7.604  ops/s
[info] SemaphoreBenchmark.semaphoreContention            10   1000  thrpt   25  101.732 ± 5.197  ops/s
[info] SemaphoreBenchmark.tsemaphoreContention           10   1000  thrpt   25  114.949 ± 5.291  ops/s

feature branch

[info] Result "zio.stm.SemaphoreBenchmark.semaphoreCatsContention":
[info]   145.410 ±(99.9%) 7.057 ops/s [Average]
[info]   (min, avg, max) = (121.232, 145.410, 158.190), stdev = 9.421
[info]   CI (99.9%): [138.353, 152.467] (assumes normal distribution)

[info] Result "zio.stm.SemaphoreBenchmark.semaphoreContention":
[info]   106.594 ±(99.9%) 3.541 ops/s [Average]
[info]   (min, avg, max) = (96.782, 106.594, 115.379), stdev = 4.727
[info]   CI (99.9%): [103.053, 110.135] (assumes normal distribution)

[info] Result "zio.stm.SemaphoreBenchmark.tsemaphoreContention":
[info]   99.766 ±(99.9%) 27.598 ops/s [Average]
[info]   (min, avg, max) = (65.356, 99.766, 152.758), stdev = 36.843
[info]   CI (99.9%): [72.168, 127.364] (assumes normal distribution)

[info] # Run complete. Total time: 00:25:07
[info] REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
[info] why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
[info] experiments, perform baseline and negative tests that provide experimental control, make sure
[info] the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
[info] Do not assume the numbers tell you what you want them to tell.
[info] Benchmark                                   (fibers)  (ops)   Mode  Cnt    Score    Error  Units
[info] SemaphoreBenchmark.semaphoreCatsContention        10   1000  thrpt   25  145.410 ±  7.057  ops/s
[info] SemaphoreBenchmark.semaphoreContention            10   1000  thrpt   25  106.594 ±  3.541  ops/s
[info] SemaphoreBenchmark.tsemaphoreContention           10   1000  thrpt   25   99.766 ± 27.598  ops/s

the result is that average ops/s on zio.stm.SemaphoreBenchmark.semaphoreCatsContention and zio.stm.SemaphoreBenchmark.semaphoreContention got worse for about 5 ops/s as expected, and average ops/s on zio.stm.SemaphoreBenchmark.tsemaphoreContention unexpectedly got better for about 15 ops/s

…io#803)

…er executing some op with lock (zio#803)

…tion (zio#803)

ghostdogpr · 2019-08-03T09:22:26Z

Great work! Could you move the STM tests from jvm to shared? That would make them run with ScalaJS and validate the compatibility.

t5kaji · 2019-08-03T10:01:15Z

I moved the STM tests from jvm to shared.
But when I ran the tests, then I got errors like this :

[error] Referring to non-existent method java.util.concurrent.CountDownLatch.<init>(scala.Int)
[error]   called from zio.stm.STMSpec.$$anonfun$e28$1()zio.ZIO
[error]   called from zio.stm.STMSpec.e28()org.specs2.matcher.MatchResult
[error]   called from zio.stm.STMSpec.$$anonfun$is$27()org.specs2.matcher.MatchResult
[error]   called from zio.stm.STMSpec.$$anonfun$is$1()org.specs2.specification.core.Fragments
[error]   called from zio.stm.STMSpec.is()org.specs2.specification.core.SpecStructure
[error]   called from org.specs2.specification.core.SpecificationStructure.$$anonfun$structure$1(org.specs2.specification.core.Env)org.specs2.specification.core.SpecStructure
[error]   called from zio.FunctionIOSpec.$$anonfun$structure$1(org.specs2.specification.core.Env)org.specs2.specification.core.SpecStructure
[error]   called from org.specs2.specification.core.SpecificationStructure.structure()scala.Function1
[error]   called from org.specs2.Specification.org$specs2$specification$core$ImmutableSpecificationStructure$$super$structure()scala.Function1
[error]   called from org.specs2.specification.core.ImmutableSpecificationStructure.$$anonfun$structure$1(org.specs2.specification.core.Env)org.specs2.specification.core.SpecStructure
[error]   called from zio.FunctionIOSpec.$$anonfun$structure$1(org.specs2.specification.core.Env)org.specs2.specification.core.SpecStructure
[error]   called from org.specs2.specification.core.ImmutableSpecificationStructure.structure()scala.Function1
[error]   called from org.specs2.Specification.structure()scala.Function1
[error]   called from org.specs2.runner.SbtTask.$$anonfun$createSpecStructure$2(org.specs2.specification.core.Env,org.specs2.specification.core.SpecificationStructure)scala.Option
[error]   called from org.specs2.runner.SbtTask.createSpecStructure(sbt.testing.TaskDef,java.lang.ClassLoader,org.specs2.specification.core.Env)org.specs2.control.eff.Eff
[error]   called from org.specs2.runner.SbtTask.executeFuture(sbt.testing.EventHandler,[sbt.testing.Logger)scala.concurrent.Future
[error]   called from org.specs2.runner.SbtTask.execute(sbt.testing.EventHandler,[sbt.testing.Logger,scala.Function1)scala.Unit
[error]   called from org.scalajs.testinterface.internal.Bridge$.$$anonfun$executeFun$1(sbt.testing.Runner,scala.Int,org.scalajs.testcommon.ExecuteRequest)scala.concurrent.Future
[error]   called from org.scalajs.testinterface.internal.Bridge$.executeFun(scala.Int,sbt.testing.Runner)scala.Function1
[error]   called from org.scalajs.testinterface.internal.Bridge$.$$anonfun$createRunnerFun$1(scala.Boolean,org.scalajs.testcommon.RunnerArgs)scala.Unit
[error]   called from org.scalajs.testinterface.internal.Bridge$.createRunnerFun(scala.Boolean)scala.Function1
[error]   called from org.scalajs.testinterface.internal.Bridge$.start()scala.Unit
[error]   called from org.scalajs.testinterface.internal.Bridge$.__exportedInits
[error]   exported to JavaScript with @JSExport
[error] involving instantiated classes:
[error]   zio.stm.STMSpec
[error]   zio.FunctionIOSpec
[error]   zio.internal.OneShotSpec
[error]   zio.PromiseSpec
[error]   zio.FiberLocalSpec
[error]   zio.duration.DurationSyntaxSpec
[error]   zio.RetrySpec
[error]   zio.RepeatSpec
[error]   zio.RefMSpec
[error]   zio.internal.ExecutorSpec
[error]   zio.ParallelErrorsSpec
[error]   zio.internal.MutableConcurrentQueueSpec
[error]   zio.FiberRefSpec
[error]   zio.RefSpec
[error]   zio.duration.DurationSpec
[error]   zio.FiberSpec
[error]   zio.internal.StackBoolSpec
[error]   zio.QueueSpec
[error]   zio.ExitSpec
[error]   org.specs2.runner.SbtTask
[error]   org.scalajs.testinterface.internal.Bridge$
.
.
.

It seems that CountDownLatch use in STMSpec causes these errors. Please teach me if you know some workaround to this.

t5kaji · 2019-08-03T11:02:39Z

I got away with CountDownLatch by replacing it with Promise.
But then I got new errors like the following:

[error] java.util.concurrent.ExecutionException: Boxed Error

I'm trying to fix this.

ghostdogpr · 2019-08-03T11:16:19Z

@heraklos we could use a Promise[Nothing, Unit] instead of CountDownLatch but then latch.countDown() would have to be unsafeRun(latch.succeed(())) so that it is "unsafely" executed.

For example:

  def e21 =
    unsafeRun {
      for {
        latch <- Promise.make[Nothing, Unit]
        done  <- Promise.make[Nothing, Unit]
        tvar1 <- TRef.makeCommit(0)
        tvar2 <- TRef.makeCommit("Failed!")
        fiber <- (STM.atomically {
                  for {
                    v1 <- tvar1.get
                    _  <- STM.succeedLazy(unsafeRun(latch.succeed(())))
                    _  <- STM.check(v1 > 42)
                    _  <- tvar2.set("Succeeded!")
                    v2 <- tvar2.get
                  } yield v2
                } <* done.succeed(())).fork
        _    <- latch.await
        old  <- tvar2.get.commit
        _    <- tvar1.set(43).commit
        _    <- done.await
        newV <- tvar2.get.commit
        join <- fiber.join
      } yield (old must_=== "Failed!") and (newV must_=== join)
    }

ghostdogpr · 2019-08-03T11:16:47Z

Oops, you found it already. Are you getting those errors only with ScalaJS?

t5kaji · 2019-08-03T11:31:33Z

@ghostdogpr
Thank you for helping me about CountDownLatch :)

And yes to your question. The error pops out only when executing ScalaJS test.
I noticed that there is

[error] Caused by: java.lang.Error: Cannot block for result to be set in JavaScript

in errors shown in the stack trace, and this message seems to come from OneShot.get
Is that something related to unsafeRunSync?

ghostdogpr · 2019-08-03T12:06:35Z

Maybe those 3 tests that target concurrent behavior should stay in JVM since they don't make too much sense in JS?

@jdegoes what do you think?

jdegoes · 2019-08-03T12:12:57Z

The whole STM test suite should run on Scala.js without issue. But unsafeRunSync cannot be used.

Specs 2 supports returning Futures, so we need to unsafeRunToFuture and tap into the Specs 2 support.

Does that make sense?

t5kaji · 2019-08-03T12:33:43Z

Thank you for your guide!
I'll replace all the tests that fail due to unsafeRunSync with unsafeRunToFuture and spec2 Future support 👍

t5kaji · 2019-08-04T01:26:32Z

@ghostdogpr
Now I got an error

[error] java.lang.Exception: TIMEOUT: 60 seconds
[error] 	at zio.BaseCrossPlatformSpec$$anon$1.run(BaseCrossPlatformSpec.scala:67)
[error] 	at java.util.TimerThread.mainLoop(Timer.java:555)
[error] 	at java.util.TimerThread.run(Timer.java:505)
[error] Error: Total 626, Failed 0, Errors 1, Passed 625
[error] Error during tests:
[error] 	zio.QueueSpec
[error] (coreTestsJVM / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 174 s, completed Aug 4, 2019 1:14:06 AM

I think my changes does not affect QueueSpec but I'm not sure of it...
Could you run again CI tests for me?

ghostdogpr · 2019-08-04T01:31:40Z

Yeah it’s probably the same issue as #1216. I restarted it.

t5kaji · 2019-08-04T01:39:18Z

Thanks a lot 😆

But another flaky test popped up... I'm sure this is the same issue as #1023

[error]     x supervise fibers in supervised
[error]  TIMEOUT: 60 seconds (TestRuntime.scala:18)

Would you run tests again?

ghostdogpr · 2019-08-04T01:45:28Z

It’s green now :D

core-tests/shared/src/test/scala/zio/stm/STMSpec.scala

ghostdogpr · 2019-08-04T03:54:53Z

LGTM, @jdegoes you wanna have a final look?

jdegoes · 2019-08-04T13:33:27Z

core/shared/src/main/scala/zio/stm/STM.scala

-
-                try if (isValid(journal)) commitJournal(journal) else loop = true
-                finally globalLock.release()
+                globalLock.synchronized {


I thought .synchronized threw exceptions on Scala.js. No???

Yes, you're right. java built-in synchronized throws exception on Scala.js.
So, I implemented and used synchronized method on LockedRef class which wraps the procedure in {} code with locking/unlocking.

@heraklos How about this:

// JS object Sync { def apply[A](anyRef: AnyRef)(f: => A): A = f // JVM object Sync { def apply[A](anyRef: AnyRef)(f: => A): A = anyRef.synchronized { f } }

Then you can replace x.synchronized { f } with Sync(x) { f }

Sound good?

@jdegoes
That sounds perfect to me!
And with Sync, STMSpec passed for both jvm/js!

The only obstacle here is the following error

parameter value anyRef in method apply is never used

for Sync of js.

// JS object Sync { def apply[A](anyRef: AnyRef)(f: => A): A = f }

I'm gonna think of some workaround.

jdegoes · 2019-08-04T13:38:06Z

core/shared/src/main/scala/zio/internal/LockedRef.scala

+
+  def lock(): Unit = {
+    var loop = true
+    while (loop) {


This spin loop is not efficient. You should just use an ordinary Java Semaphore or Lock. An alternative would be to implement a lock atop synchronized, which would be more efficient than anything else. You have to know how to use waitAll / notifyAll and deal with spurious wakeups, etc. So the simple solution is to use Semaphore here the more complex one is to build something low-level

Thank you for your advice on locking!
Unfortunately, java.util.concurrent.Semaphore is not supported by scala.js...
Maybe I should implement lock with ReentrantLock here while learning waitAll / notifyAll and other concurrent matter to be cared for, since I'm not familiar with those concepts. (java's built-in synchronized is not supported as you taught us in #803)

But on reading your overall review on this PR, this class is not needed in the first place...?

Javascript will not need a semaphore. I think we can get by with purely synchronized (and Sync class above), without any waiting / awaiting.

Can you give it a try and let me know? Thank you!

Yes, you're totally correct.
The mechanism of Javascript runtime slipped my mind.

jdegoes · 2019-08-04T13:38:43Z

core/shared/src/main/scala/zio/internal/LockedRef.scala

+
+import java.util.concurrent.atomic.AtomicBoolean
+
+class LockedRef[A](var ref: A) {


If this raw var is stored here, it needs to be marked as @volatile.

jdegoes · 2019-08-04T13:46:04Z

core/shared/src/main/scala/zio/stm/STM.scala

@@ -561,7 +570,7 @@ object STM {

    private[this] val txnCounter: AtomicLong = new AtomicLong()

-    final val globalLock = new java.util.concurrent.Semaphore(1)
+    final val globalLock = new zio.internal.LockedRef(1)


This is not even using the Int passed into LockedRef, which means globalLock is unnecessary. You may as well just synchronize on STM object, e.g. STM.synchronized { ... } or maybe make a private object, e.g. final val globalLock = new AnyRef { }.

You're right! This LockedRef instance does not use the passed Int at all.
but my understanding is that we can't use java's built-in synchronized in scala.js.
So, I have to come up with some data structure similar to LockedRef which does not take an argument.

jdegoes

Thank you for your work on this! Supporting Scala.js is very critical to 1.0 and this moves us a long way there.

Upon review of this pull request, I don't think we need LockedRef.scala. Indeed, no methods are used, it's just used as a container.

Instead, I think we can get by simple with AtomicBoolean + synchronized for done, and a val globalLock: AnyRef = new AnyRef { } for the global lock (using synchronized like you are currently doing).

This will greatly reduce code and improve performance and should be a quick change to make.

Let me know if you have any questions!

t5kaji · 2019-08-04T16:16:47Z

@jdegoes
Thank you for your detailed review 😆
Your advice seems perfectly reasonable to me and I would like to implement that.

But here is the problem. I actually used LockedRef for synchronizing some operations which are synchronized by java's built-in synchronized before. So, in my opinion, it's not just a container class.

So I'm afraid that we can get by simple with AtomicBoolean + synchronized for done, and a val globalLock: AnyRef = new AnyRef { } for the global lock as you recommended for me, because synchronized is LockedRef's method (sorry for confusing method name)

Instead, what about if separating LockedRef into jvm/js package and implement lock/unlock with Semaphore in jvm side, and implement lock/unlock with Lock or other possible solution in js side for better performance?

t5kaji · 2019-08-05T02:13:19Z

core/js/src/main/scala/zio/internal/Sync.scala

+package zio.internal
+
+object Sync {
+  def apply[A](anyRef: AnyRef)(f: => A): A = {


For STM.scala to be shared, this method should take the same arguments as that of Sync in jvm does.
But the compiler does not allow us to take anyRef and does nothing with it.
So I avoided the error just by asserting the existence of anyRef.

You can do, inside the body:

val _ = anyRef

jdegoes

Excellent work! Thank you for your patience on this large PR!

jdegoes · 2019-08-06T09:01:11Z

ghostdogpr · 2019-08-06T09:04:15Z

@heraklos Glad to see you went through this 👏👏👏 One less blocker for ZIO 1.0!

t5kaji · 2019-08-06T09:35:18Z

@jdegoes
Thanks a lot for teaching me 😆
I'm really glad that this PR was merged since this was my first contribution to application code in zio!
Next time, I'll do it more quickly and nicely :)

@ghostdogpr
Thank you so much for your warmful support 😆I finally made it!
Your continuous help enabled me to move forward at all times. Thanks!

jdegoes · 2019-08-06T09:57:09Z

@heraklos Congratulations on your first merge to core... first of many, I hope! 😉

heraklos added 8 commits July 20, 2019 10:06

Replace synchronized block with LockedRef data type (zio#803)

d1a9516

Fix locking mechanism of LockedRef (zio#803)

e3c8e33

Merge remote-tracking branch 'upstream/master' into port-stm-to-scalajs

3c82769

Fix locking logic flaw in LockedRef (zio#803)

0cc7091

Replace Semaphore for controlling commit to journal with LockedRef (z…

85ab30c

…io#803)

Merge remote-tracking branch 'upstream/master' into port-stm-to-scalajs

c37d7e1

Introduce synchronized method in LockedRef to make sure to unlock aft…

ed6edd3

…er executing some op with lock (zio#803)

Replace jdk8 higher-order method calls with iterator-based implementa…

b801f8d

…tion (zio#803)

t5kaji changed the title ~~[WIP] stm support scala.js~~ [WIP] stm supports scala.js Aug 3, 2019

t5kaji changed the title ~~[WIP] stm supports scala.js~~ STM supports scala.js Aug 3, 2019

t5kaji mentioned this pull request Aug 3, 2019

STM: Support Scala.js #803

Closed

t5kaji force-pushed the port-stm-to-scalajs branch from ea79bcb to 76dee8e Compare August 3, 2019 11:46

t5kaji force-pushed the port-stm-to-scalajs branch from 76dee8e to 639fdf8 Compare August 3, 2019 16:20

Move STMSpec from jvm to shared test (zio#803)

e6eeca5

t5kaji force-pushed the port-stm-to-scalajs branch from 639fdf8 to e6eeca5 Compare August 4, 2019 01:07

ghostdogpr reviewed Aug 4, 2019

View reviewed changes

core-tests/shared/src/test/scala/zio/stm/STMSpec.scala Show resolved Hide resolved

Remove unsafeRun test executions from STMSpec for consistency (zio#803)

30c3b30

ghostdogpr previously approved these changes Aug 4, 2019

View reviewed changes

jdegoes reviewed Aug 4, 2019

View reviewed changes

jdegoes previously approved these changes Aug 4, 2019

View reviewed changes

Replace LockedRef with Sync provided for each jvm/js (zio#803)

4690613

t5kaji dismissed stale reviews from jdegoes and ghostdogpr via 4690613 August 5, 2019 02:06

t5kaji commented Aug 5, 2019

View reviewed changes

Make it possible to compile for scala.js (zio#803)

eaa0976

jdegoes approved these changes Aug 6, 2019

View reviewed changes

jdegoes merged commit a941ed8 into zio:master Aug 6, 2019

t5kaji deleted the port-stm-to-scalajs branch August 7, 2019 09:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

STM supports scala.js #1338

STM supports scala.js #1338

t5kaji commented Aug 3, 2019 •

edited

ghostdogpr commented Aug 3, 2019

t5kaji commented Aug 3, 2019 •

edited

t5kaji commented Aug 3, 2019

ghostdogpr commented Aug 3, 2019

ghostdogpr commented Aug 3, 2019

t5kaji commented Aug 3, 2019 •

edited

ghostdogpr commented Aug 3, 2019

jdegoes commented Aug 3, 2019

t5kaji commented Aug 3, 2019

t5kaji commented Aug 4, 2019

ghostdogpr commented Aug 4, 2019

t5kaji commented Aug 4, 2019

ghostdogpr commented Aug 4, 2019

ghostdogpr commented Aug 4, 2019

jdegoes Aug 4, 2019

t5kaji Aug 4, 2019 •

edited

jdegoes Aug 4, 2019

t5kaji Aug 5, 2019

jdegoes Aug 4, 2019

t5kaji Aug 4, 2019 •

edited

jdegoes Aug 4, 2019

t5kaji Aug 5, 2019

jdegoes Aug 4, 2019

jdegoes Aug 4, 2019

t5kaji Aug 4, 2019

jdegoes left a comment

t5kaji commented Aug 4, 2019 •

edited

t5kaji Aug 5, 2019 •

edited

jdegoes Aug 5, 2019

jdegoes left a comment

jdegoes commented Aug 6, 2019

ghostdogpr commented Aug 6, 2019

t5kaji commented Aug 6, 2019

jdegoes commented Aug 6, 2019


		import java.util.concurrent.atomic.AtomicBoolean

		class LockedRef[A](var ref: A) {

STM supports scala.js #1338

STM supports scala.js #1338

Conversation

t5kaji commented Aug 3, 2019 • edited

Changes in code

Changes in performance

ghostdogpr commented Aug 3, 2019

t5kaji commented Aug 3, 2019 • edited

t5kaji commented Aug 3, 2019

ghostdogpr commented Aug 3, 2019

ghostdogpr commented Aug 3, 2019

t5kaji commented Aug 3, 2019 • edited

ghostdogpr commented Aug 3, 2019

jdegoes commented Aug 3, 2019

t5kaji commented Aug 3, 2019

t5kaji commented Aug 4, 2019

ghostdogpr commented Aug 4, 2019

t5kaji commented Aug 4, 2019

ghostdogpr commented Aug 4, 2019

ghostdogpr commented Aug 4, 2019

Choose a reason for hiding this comment

t5kaji Aug 4, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t5kaji Aug 4, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdegoes left a comment

Choose a reason for hiding this comment

t5kaji commented Aug 4, 2019 • edited

t5kaji Aug 5, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdegoes left a comment

Choose a reason for hiding this comment

jdegoes commented Aug 6, 2019

ghostdogpr commented Aug 6, 2019

t5kaji commented Aug 6, 2019

jdegoes commented Aug 6, 2019

t5kaji commented Aug 3, 2019 •

edited

t5kaji commented Aug 3, 2019 •

edited

t5kaji commented Aug 3, 2019 •

edited

t5kaji Aug 4, 2019 •

edited

t5kaji Aug 4, 2019 •

edited

t5kaji commented Aug 4, 2019 •

edited

t5kaji Aug 5, 2019 •

edited