Enable parallel optimizing and writing of classes by GenBCode #6124

mkeskells · 2017-10-11T18:27:57Z

Introduces a compiler flag -Ybackend-parallelism N which allows specifying the number of threads for optimizing and writing classes in parallel.

The granularity of parallelization is a compilation unit (a source file).

We measured speedup of 4-5% when compiling the scala/scala sources with 8 backend threads (graph).

rework ClassBType to enable parallelism, and move logic in the companion
rewrite ClasfileWriter, specialising for JAR/dir, and providing wrappers for the less common cases
rework directory classfile writing to be async NIO based.
Async the directory creation
make PerRunInit theadsafe

BackendUtils - make some data structure/APIs theadsafe (indyLambdaMethods)
add extra parameter -YmaxAdditionalWriterThreads .. "maximum additional threads to write class files"
add a class handler as a delegate that allows the minimal set in post-processing steps to be performed

mkeskells · 2017-10-11T18:46:37Z

here are some stats for this change, run prior to last rebase, and should be close to current - I will generate more stats for a run later with the new baseline and rebased code

Configuration -
test is performed on a 4 core I7 (2 rel core + hyperthreading) windows 10 system
baseline is the baseline commit, other lines relate to the new code with varying settings of -YmaxAdditionalWriterThreads

each test compile is run 20 times in the same VM to warm up

all timings are in ms.The second colun is not so interesting - number of phases included

ALL - refers to all runs - time is for the full compile
after 10 90% - ignores the first 10 runs, and the worse 10 % of results ( e.g. heavy GC) - time is for the full compile
after 10 90%, phase jvm, no GC - as above but limits the timing to the jvm phase ( this PR is targetted at the jvm phase, and makes no material change outside that phase), and disguards measurement including a GC

ALL

                  RunName	                AllWallMS	        
      00_backend-baseline	 810	12,148.60 [-22.10% +185.53%]
             00_backend-0	 810	12,144.77 [-19.87% +220.26%]
             00_backend-1	 810	11,817.25 [-18.54% +204.61%]
             00_backend-2	 810	11,353.86 [-23.45% +194.46%]
             00_backend-3	 810	10,224.86 [-20.78% +229.95%]
             00_backend-4	 810	10,190.22 [-21.00% +225.74%]
             00_backend-5	 810	9,808.82 [-22.97% +233.42%]	
             00_backend-6	 810	9,804.54 [-23.69% +229.63%]	
             00_backend-7	 810	9,559.68 [-22.52% +238.89%]	
             00_backend-8	 810	9,473.14 [-23.06% +260.49%]	
after 10 90%

                  RunName	                AllWallMS	        
      00_backend-baseline	 510	10,157.17 [-6.82% +7.54%]	
             00_backend-0	 510	10,180.64 [-4.41% +2.87%]	
             00_backend-1	 510	 9,841.78 [-2.18% +2.58%]	
             00_backend-2	 510	 8,959.68 [-2.99% +4.71%]	
             00_backend-3	 510	 8,424.00 [-3.84% +2.40%]	
             00_backend-4	 510	 8,314.82 [-3.18% +2.75%]	
             00_backend-5	 510	 7,886.69 [-4.20% +3.23%]	
             00_backend-6	 510	 7,868.82 [-4.92% +3.58%]	
             00_backend-7	 510	 7,606.62 [-2.62% +3.20%]	
             00_backend-8	 510	 7,487.36 [-2.66% +3.35%]	
after 10 90%, phase jvm, no GC

                  RunName	                AllWallMS	        
      00_backend-baseline	  16	 3,685.55 [-5.46% +7.87%]	
             00_backend-0	  15	 3,481.92 [-3.84% +4.13%]	
             00_backend-1	  16	 3,394.66 [-4.64% +5.73%]	
             00_backend-2	  18	 2,471.76 [-8.64% +6.61%]	
             00_backend-3	  17	 2,096.19 [-5.71% +4.60%]	
             00_backend-4	  18	1,825.50 [-6.95% +10.41%]	
             00_backend-5	  17	1,491.01 [-14.57% +7.66%]	
             00_backend-6	  16	1,304.25 [-15.29% +13.13%]	
             00_backend-7	  15	1,165.13 [-7.80% +14.10%]	
             00_backend-8	  16	 1,118.78 [-6.36% +6.55%]

memory allocation (in GenBCode) are reduced slightly from 423.26 MB to 412.32 Mb for single threaded. MT figures are not available due to lack of instrumentation

mkeskells · 2017-10-11T18:55:07Z

This PR is not complete
e.g. It needs rerun of performance on UNIX/Mac, and there is still some tidyup to do and it need a validation of the additional code paths introduced, but as it provides a measurable performance improvement even in the single threaded case, and a marked throughput in the MT case, I thought that it was worth pushing the review a little early

mkeskells · 2017-10-11T19:10:01Z

Memory pressure is reduced

the processing is pipelined, with the later phases ( writing the data to disk) recovering the memory as soon as the file is written)
the pipeline is limited, and if the writer cant keep up, the early phase has back-pressure

lrytz · 2017-10-11T20:31:39Z

Exciting! I'll take a look on Friday.

mkeskells · 2017-10-12T18:51:40Z

Updated some more extensive testing results - same windows machine, but 150 iterations of each compile so that the number are more stable.

ALL
                  RunName	                AllWallMS	        
      00_backend-baseline	4050	10,222.16 [-8.60% +291.22%]	
             00_backend-0	4050	9,874.56 [-8.87% +180.48%]	
             00_backend-1	4050	7,823.89 [-11.05% +335.20%]	
             00_backend-2	4050	7,747.43 [-11.59% +330.97%]	
             00_backend-3	4050	7,891.42 [-13.35% +326.03%]	
             00_backend-4	4050	7,758.51 [-11.48% +338.29%]	
             00_backend-6	4050	7,904.31 [-11.43% +331.10%]	
             00_backend-8	4050	7,963.89 [-11.47% +323.16%]	
            00_backend-10	4050	7,745.32 [-11.52% +329.60%]	
            00_backend-12	4050	7,825.48 [-13.25% +324.15%]	
            00_backend-14	4050	7,814.28 [-10.58% +327.63%]	
            00_backend-16	4050	7,733.57 [-11.01% +323.25%]	
after 10 90%
                  RunName	                AllWallMS	        
      00_backend-baseline	3426	 9,728.14 [-3.96% +4.95%]	
             00_backend-0	3426	 9,458.36 [-4.86% +7.03%]	
             00_backend-1	3426	 7,366.27 [-5.52% +6.12%]	
             00_backend-2	3426	 7,303.72 [-6.22% +5.00%]	
             00_backend-3	3426	 7,379.21 [-7.34% +5.81%]	
             00_backend-4	3426	 7,284.50 [-5.72% +5.69%]	
             00_backend-6	3426	 7,458.70 [-6.14% +5.16%]	
             00_backend-8	3426	 7,517.85 [-6.21% +4.45%]	
            00_backend-10	3426	 7,278.99 [-5.85% +7.05%]	
            00_backend-12	3426	 7,327.12 [-7.35% +7.02%]	
            00_backend-14	3426	 7,364.65 [-5.12% +5.44%]	
            00_backend-16	3426	 7,265.07 [-5.27% +5.65%]	
after 10 90%, phase jvm, no GC
                  RunName	                AllWallMS	        
      00_backend-baseline	 117	 3,867.26 [-3.81% +4.71%]	
             00_backend-0	 117	 3,244.24 [-5.06% +7.08%]	
             00_backend-1	 112	1,344.32 [-12.66% +9.37%]	
             00_backend-2	 117	1,355.05 [-15.18% +9.20%]	
             00_backend-3	  73	1,337.02 [-13.99% +11.27%]	
             00_backend-4	 125	1,351.97 [-17.44% +9.70%]	
             00_backend-6	  67	1,337.12 [-12.08% +9.96%]	
             00_backend-8	  76	1,319.22 [-14.54% +10.70%]	
            00_backend-10	 120	1,343.10 [-12.30% +8.96%]	
            00_backend-12	  95	1,325.05 [-16.23% +11.20%]	
            00_backend-14	 125	1,322.21 [-11.65% +10.53%]	
            00_backend-16	 121	1,325.99 [-16.11% +10.90%]	
after 20 90%
                  RunName	                AllWallMS	        
      00_backend-baseline	3183	 9,702.79 [-3.71% +4.44%]	
             00_backend-0	3183	 9,417.31 [-4.45% +6.09%]	
             00_backend-1	3183	 7,336.39 [-5.14% +6.02%]	
             00_backend-2	3183	 7,279.51 [-5.90% +5.15%]	
             00_backend-3	3183	 7,357.65 [-7.07% +6.01%]	
             00_backend-4	3183	 7,259.71 [-5.40% +5.47%]	
             00_backend-6	3183	 7,438.31 [-5.88% +4.78%]	
             00_backend-8	3183	 7,501.33 [-6.01% +4.42%]	
            00_backend-10	3183	 7,252.99 [-5.52% +6.73%]	
            00_backend-12	3183	 7,305.25 [-7.07% +7.34%]	
            00_backend-14	3183	 7,341.15 [-4.82% +5.65%]	
            00_backend-16	3183	 7,236.57 [-4.90% +5.35%]	
after 20 90%, phase jvm, no GC
                  RunName	                AllWallMS	        
      00_backend-baseline	 108	 3,856.10 [-3.53% +4.33%]	
             00_backend-0	 108	 3,228.20 [-4.59% +7.02%]	
             00_backend-1	 104	1,336.48 [-12.15% +9.06%]	
             00_backend-2	 108	1,348.77 [-14.79% +9.54%]	
             00_backend-3	  65	1,325.20 [-13.22% +10.45%]	
             00_backend-4	 116	1,345.73 [-17.06% +9.49%]	
             00_backend-6	  61	1,328.12 [-11.48% +10.16%]	
             00_backend-8	  69	1,307.97 [-13.80% +11.47%]	
            00_backend-10	 113	1,338.55 [-12.00% +8.88%]	
            00_backend-12	  87	1,316.63 [-15.69% +11.56%]	
            00_backend-14	 116	1,317.76 [-11.35% +10.81%]	
            00_backend-16	 113	1,318.13 [-15.61% +9.09%]

mkeskells · 2017-10-13T07:28:43Z

update - mostly tidup-ups

combine sync and async writers code path
make back-pressure work - this will affect timings
simplify asmp and dump writers, combining with DirWriter
preserve old exception handling model
use java based jar writing

as the back pressure fix will significantly affect timing I will re-run and update the previous published timings, when ready. It takes a few hours of CPU to do this though

lrytz · 2017-10-13T08:45:20Z

It seems the the individual changes you list in the commit messages seem to be independent of each other, so they should be in separate commits. Having commits focused on one topic saves a lot of time when going through the repo history (in the future).

lrytz

I haven't looked at everything in all details, in particular the third commit. In general it looks very good! I think there is a bit of over-abstraction in ClassfileWriter / ClassHandler / UnitInfoLookup and their subclasses. I'll look at it more later, maybe some of the things can be simplified.

lrytz · 2017-10-13T09:06:02Z

src/compiler/scala/tools/nsc/backend/jvm/BTypesFromSymbols.scala

@@ -261,7 +261,7 @@ abstract class BTypesFromSymbols[G <: Global](val global: G) extends BTypes {
      r
  })(collection.breakOut)

-  private def setClassInfo(classSym: Symbol, classBType: ClassBType): ClassBType = {
+  private def computeJavaClassInfo(classSym: Symbol, classBType: ClassBType): Right[Nothing, ClassInfo] = {


This method is not only used for java-defined classes, so it should be named computeClassInfo.

done. I have no memory as to why I changed it ...

lrytz · 2017-10-13T09:50:32Z

src/compiler/scala/tools/nsc/backend/jvm/BTypes.scala

+    def unapply(cr:ClassBType) = Some(cr.internalName)
+
+    def apply(internalName: InternalName, cache: mutable.Map[InternalName, ClassBType])(init: (ClassBType) => Either[NoClassBTypeInfo, ClassInfo]) = {
+      assert (Thread.holdsLock(frontendAccess.frontendLock))


This is not obvious to me. Before this PR, we frontendLock was guarding accesses to the frontend, and the Lazy class used in ClassInfo. IUC, this PR puts CodeGen.genClassDef under that lock, too. But I think there are other places where ClassBTypes are created, for example during inlining (classBTypeFromParsedClassfile). Did you test running the parallel backend with the optimizer enabled?

lrytz · 2017-10-13T10:11:12Z

src/compiler/scala/tools/nsc/backend/jvm/BTypes.scala

+    def apply(internalName: InternalName, cache: mutable.Map[InternalName, ClassBType])(init: (ClassBType) => Either[NoClassBTypeInfo, ClassInfo]) = {
+      assert (Thread.holdsLock(frontendAccess.frontendLock))
+      val res = new ClassBType(internalName)
+      res.synchronized {


So we want to ensure that there is a single ClassBType created for each InternalName, also if there are multiple threads. IUC the synchronization here (and within def info above) is to ensure that no thread sees a partially constructed instance.

But I think in the current state of this PR, there could still be a race condition. If two threads invoke classBTypeFromParsedClassfile for the same InternalName, both could get None from cachedClassBType and create a new instance.

lrytz · 2017-10-13T11:49:45Z

src/compiler/scala/tools/nsc/backend/jvm/opt/ByteCodeRepository.scala

@@ -67,6 +67,7 @@ abstract class ByteCodeRepository {
   * of a ClassNode is about 30 kb. I observed having 17k+ classes in the cache, i.e., 500 mb.
   */
  private def limitCacheSize(): Unit = {
+    println(s"limitCacheSize ${parsedClasses.size} $maxCacheSize")


lrytz · 2017-10-13T12:10:05Z

src/compiler/scala/tools/nsc/backend/jvm/PerRunInit.scala

@@ -9,10 +9,13 @@ import scala.collection.mutable.ListBuffer
 * The trait provides an `initialize` method that runs all initializers added through `perRunLazy`.
 */
 trait PerRunInit {
+  //We have to synchronize on inits, as many of the initialisers are themselves lazy,
+  // so the back end may initialise them in parallel, and ListBuffer is not threadsafe
+  // Not sure it is sensible to have a perRunInit that is itself a lazy val


the idea is that only those PerRunInits that are actually used are also created, and re-initialized on the next run.

yes but we have lazy val which are per run init, so the registration of them is potentially in multiple threads, either at class load time or when the val is first touched. I am not sure why a per-run is lazy anyway, but I could not reason as to it being guaranteed to be threadsafe, and at best it would be fragile.

The problem is the actual lit f per-runs not in the actual target

lrytz · 2017-10-13T12:35:19Z

src/compiler/scala/tools/nsc/backend/jvm/GenBCode.scala

-      compilerSettings.optInlinerEnabled || compilerSettings.optClosureInvocations
-    }
-
+    private var generatedHandler:ClassHandler = _


This var field doesn't fit in the backend's design. Could this be a component of CodeGen? If there's state that needs to be re-initialized, it would be good to fit it into the perRunLazy infrastructure.

probably could be a per-run. Have you measure the overhead of per-run, or considered the option of an assigned clearable structure, that is directly accessed?

Its effectively a local variable, but because the run loops via apply to run for each CompilationUnit it needs to be shared between the 2 methods and cant be passed directly

lrytz · 2017-10-13T12:38:50Z

src/compiler/scala/tools/nsc/backend/jvm/BTypes.scala

@@ -778,6 +778,15 @@ abstract class BTypes {
      } while (fcs == null)
      fcs
    }
+
+    // equallity and hashcode is based on internalName
+    override def equals(obj: scala.Any): Boolean = obj match {


I think reference equality / identity hashCode should be fine here, no? The construction pattern should guarantee that there's a single instance per InternalName.

i did this because it failed some unit tests. It is nt used in production though,just a test that ran from a symbol and from a class file

mkeskells · 2017-10-13T22:03:00Z

/rebuild

mkeskells · 2017-10-14T07:36:19Z

found an initialization order issue which means that all of the MT results used the default scheduler with 4 threads and no back pressure. will post an update on Sunday with update in bold

this fixed done
back pressure working done
investigation on the appropriate and optimum queue size partial added -Ymax-queue
statistics capture on the background thread cpu io and idle time done
I target 1 second elapsed as the measure of success 😊 done

mkeskells · 2017-10-14T07:37:53Z

these changes should not affect the review. they affect scheduling not algo

mkeskells · 2017-10-16T07:19:42Z

latest commit has a marked perf improvement in windows IO - below is the affect in single threaded windows

             RunName	                AllWallMS	        
 00_backend-baseline	 116	 4,653.97 [-2.05% +4.33%]	
        00_backend-0	 119	 2,571.55 [-4.41% +4.94%]

other figures distorted due to too much GC, but not getting the expected parallelism. Will rerun the stats wih -Yprofile-run-gc and keep digging

mkeskells · 2017-10-23T22:03:00Z

/rebuild

mkeskells · 2017-10-23T22:04:42Z

spun out #6145 related (same phase) but not interlocking as is #6125

mkeskells · 2017-10-30T21:47:27Z

/rebuild

mkeskells · 2017-11-05T22:55:45Z

spun out some work in #6162

lrytz · 2017-11-10T15:35:46Z

@mkeskells before going deeper into reviewing the actual implementation, I'd like to work on the structure of this PR. Could you create individual commits for each topic, don't blend things that can be separated. Examples (mostly from your commit message):

rework ClassBType to enable parallelism
rework directory classfile writing to be async NIO based
refactor ClasfileWriter
make PerRunInit theadsafe
BackendUtils - make some data structure/APIs theadsafe (indyLambdaMethods and maxLocalStackComputed)
commit that actually enables parallelization; can we split it up into specific parts being
parallelized (io, asm's ClassWriter / serialization, parts of the post processing)

For each reafactoring, it would be nice to have (in the commit message, maybe in code where it makes sense) a high-level overview of the new architecture and a motivation why / how this enables future changes (parallelism).

You don't need to break things into separate PRs (only if a change makes sense individually, but things that depend on each other better stay together), but the commits should be on one topic and self-contained.

Even before doing that, it would be helpful if you could lay out the main plan that you have in mind for parallelizing the backend, and the observations that you made so far.

mkeskells · 2018-02-16T09:35:58Z

/rebuild

lrytz · 2018-02-16T12:37:18Z

Somehow github shows my comments collapsed, but the two remaining nits are

mkeskells · 2018-02-16T15:42:54Z

Hmm - that irratating - will try to do it properly ....

lrytz · 2018-02-19T08:31:12Z

@mkeskells to summarize: I took the branch from my second review pass, the one you squashed into a single commit (c8e6887). I rebased my branch to make sure it has the same baseline as your PR. The diff of those two branches is this: https://gist.github.com/lrytz/5b0db9dd3829bb15140f098161fe6fc5, so they are almost the same. The only things that i'd still like to include from that diff

make CompilationUnitPaths a component (https://gist.github.com/lrytz/5b0db9dd3829bb15140f098161fe6fc5#file-pr6124-diff-L127)
remove the statistics parameter from WithStatsWriter, use unsafeStatistics instead (https://gist.github.com/lrytz/5b0db9dd3829bb15140f098161fe6fc5#file-pr6124-diff-L27)

mkeskells · 2018-02-19T08:49:08Z

@lrytz pushed untested changes for review

lrytz · 2018-02-19T16:05:30Z

/synch

lrytz · 2018-02-19T16:07:15Z

ah well, scabot is still familiarizing with the new CI setup..

lrytz

🚀

rorygraves · 2018-02-19T16:14:16Z

Woot!

hrhino · 2018-02-19T16:21:37Z

🎉

mkeskells · 2018-02-19T23:11:48Z

:-)
its a long while since this started - #5815
but we snuck in before its birthday

SethTisue · 2018-02-20T00:47:34Z

very cool to see this crossing the finish line!

retronym · 2018-03-21T08:14:32Z

Minor regression under -d <symlink> fixed in #6450

som-snytt · 2019-11-30T04:27:28Z

Also under -d script.jar. scala/bug#11815

If a bug swarms in the forest and no one is there to be bothered by it, does it still annoy?

This PR ports JVM backend refactor from Scala 2 as part of the #14912 thread. It squashes changes based on the PRs: - scala/scala#6012 - scala/scala#6057 The last refactor introducing backend parallelism scala/scala#6124 is left for later.

This PR ports scala/scala#6124 introduces backend parallelism, on top of the previously ported changes introduced in #15322 Adds Scala2 flags: * `-Yjar-compression-level:N` * `-Ybackend-parallelism:N` * `-Ybackend-worker-queue:N`

scala-jenkins added this to the 2.12.5 milestone Oct 11, 2017

mkeskells mentioned this pull request Oct 11, 2017

parallelise the back end writing class files rorygraves/scalac_perf#13

Closed

mkeskells force-pushed the mike/2.12.x-backend-parallel-rebase branch from b1b0dd9 to dd3ce70 Compare October 11, 2017 22:14

lrytz reviewed Oct 13, 2017

View reviewed changes

mkeskells force-pushed the mike/2.12.x-backend-parallel-rebase branch 2 times, most recently from 3022b47 to ebdaea5 Compare November 19, 2017 22:35

This was referenced Nov 19, 2017

improve benchmarking of multi-threaded compilation #6182

Merged

use java concurrent hash map for speed in GenBCode #6183

Merged

mkeskells force-pushed the mike/2.12.x-backend-parallel-rebase branch from 5514fd6 to 1dfbb32 Compare November 19, 2017 23:55

smarter mentioned this pull request Nov 20, 2017

Consider making a TASTY backend twitter/rsc#21

Closed

mkeskells mentioned this pull request Nov 21, 2017

allow PerRunInit to operate in multi-threaded environment #6189

Merged

This was referenced Nov 29, 2017

simple memory and inlining changes #6202

Merged

optimise use of indyLambdaMethods map #6203

Merged

improve construction of ClassBTypes #6201

Merged

mkeskells force-pushed the mike/2.12.x-backend-parallel-rebase branch 2 times, most recently from 7c45bd8 to c8e6887 Compare February 18, 2018 22:06

review feedback

aa39836

lrytz approved these changes Feb 19, 2018

View reviewed changes

lrytz merged commit ea64efa into scala:2.12.x Feb 19, 2018

SethTisue mentioned this pull request Feb 23, 2018

8 threads #6346

Closed

lrytz added the release-notes worth highlighting in next release notes label Mar 14, 2018

lrytz changed the title ~~enable parallel writing of classes by GenBCode~~ Enable parallel optimizing and writing of classes by GenBCode Mar 14, 2018

lrytz added 2.12 and removed 2.12 labels Mar 14, 2018

renovate bot mentioned this pull request Jun 27, 2021

chore(deps): update dependency scala to v2.12.14 limpid-kzonix/kzonix-opencv-app#6

Merged

1 task

renovate bot mentioned this pull request Aug 11, 2021

Update dependency scala to v2.12.14 - autoclosed bgoonz/The-Algorithms#7

Closed

1 task

smarter mentioned this pull request Apr 12, 2022

Port missing changes from the Scala 2 backend (bug fixes for inner class generation, refactorings, optimizations ...) scala/scala3#14912

Closed

6 tasks

WojciechMazur mentioned this pull request May 30, 2022

Port JVM backend refactor from Scala 2 scala/scala3#15322

Merged

2 tasks

WojciechMazur mentioned this pull request Jun 7, 2022

Parallelise JVM backend - Scala 2 port scala/scala3#15392

Merged

Enable parallel optimizing and writing of classes by GenBCode #6124

Enable parallel optimizing and writing of classes by GenBCode #6124

Conversation

mkeskells commented Oct 11, 2017 • edited by lrytz Loading

mkeskells commented Oct 11, 2017 • edited Loading

mkeskells commented Oct 11, 2017

mkeskells commented Oct 11, 2017

lrytz commented Oct 11, 2017

mkeskells commented Oct 12, 2017 • edited Loading

mkeskells commented Oct 13, 2017 • edited Loading

lrytz commented Oct 13, 2017

lrytz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkeskells commented Oct 13, 2017

mkeskells commented Oct 14, 2017 • edited Loading

mkeskells commented Oct 14, 2017

mkeskells commented Oct 16, 2017 • edited Loading

mkeskells commented Oct 23, 2017

mkeskells commented Oct 23, 2017

mkeskells commented Oct 30, 2017

mkeskells commented Nov 5, 2017

lrytz commented Nov 10, 2017

mkeskells commented Feb 16, 2018

lrytz commented Feb 16, 2018

mkeskells commented Feb 16, 2018

lrytz commented Feb 19, 2018

mkeskells commented Feb 19, 2018

lrytz commented Feb 19, 2018

lrytz commented Feb 19, 2018

lrytz left a comment

Choose a reason for hiding this comment

rorygraves commented Feb 19, 2018

hrhino commented Feb 19, 2018

mkeskells commented Feb 19, 2018

SethTisue commented Feb 20, 2018

retronym commented Mar 21, 2018

som-snytt commented Nov 30, 2019

mkeskells commented Oct 11, 2017 •

edited by lrytz

Loading

mkeskells commented Oct 11, 2017 •

edited

Loading

mkeskells commented Oct 12, 2017 •

edited

Loading

mkeskells commented Oct 13, 2017 •

edited

Loading

mkeskells commented Oct 14, 2017 •

edited

Loading

mkeskells commented Oct 16, 2017 •

edited

Loading