Add Java 11 to the testsuite #577

Fokko · 2019-10-25T23:21:11Z

Add JDK11 to the Travis build matrix.

rdblue · 2019-10-27T21:46:00Z

Thanks @Fokko, this is a good idea.

Tests are failing because of the sanity check in build.gradle that requires Java 8. We will need to disable that by default, but we do want to activate it when the release property is set so that binaries are built with the oldest supported version of Java. There's an example of using that property to control configuration here: https://github.com/apache/incubator-iceberg/blob/master/build.gradle#L192

rdblue · 2019-11-01T18:22:23Z

Here's a branch that moves the Java 8 check and only enforces it for a release build: rdblue@146c22a

That allowed me to run with Java 11, but there are test failures to look into.

Fokko · 2019-11-03T19:02:02Z

Looks like there are issues with older versions of Datanucleaus (3.x branch) which ships with Hive test artifacts:

java.lang.RuntimeException: Cannot start TestHiveMetastore
	at org.apache.iceberg.hive.TestHiveMetastore.start(TestHiveMetastore.java:69)
	at org.apache.iceberg.hive.HiveMetastoreTest.startMetastore(HiveMetastoreTest.java:41)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.jdo.JDOFatalInternalException: The java type java.lang.Long (jdbc-type="", sql-type="") cant be mapped for this datastore. No mapping is available.
NestedThrowables:
org.datanucleus.exceptions.NucleusException: The java type java.lang.Long (jdbc-type="", sql-type="") cant be mapped for this datastore. No mapping is available.
	at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:591)
	at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
	at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
	at org.apache.hadoop.hive.metastore.ObjectStore.setMetaStoreSchemaVersion(ObjectStore.java:6773)
	at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6670)
	at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
	at com.sun.proxy.$Proxy16.verifySchema(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:624)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:419)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:412)
	at org.apache.iceberg.hive.TestHiveMetastore.newThriftServer(TestHiveMetastore.java:97)
	at org.apache.iceberg.hive.TestHiveMetastore.start(TestHiveMetastore.java:65)
	... 41 more
Caused by: org.datanucleus.exceptions.NucleusException: The java type java.lang.Long (jdbc-type="", sql-type="") cant be mapped for this datastore. No mapping is available.
	at org.datanucleus.store.rdbms.mapping.RDBMSMappingManager.getDatastoreMappingClass(RDBMSMappingManager.java:1215)
	at org.datanucleus.store.rdbms.mapping.RDBMSMappingManager.createDatastoreMapping(RDBMSMappingManager.java:1378)
	at org.datanucleus.store.rdbms.table.AbstractClassTable.addDatastoreId(AbstractClassTable.java:392)
	at org.datanucleus.store.rdbms.table.ClassTable.initializePK(ClassTable.java:1087)
	at org.datanucleus.store.rdbms.table.ClassTable.preInitialize(ClassTable.java:247)
	at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTable(RDBMSStoreManager.java:3118)
	at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTables(RDBMSStoreManager.java:2909)
	at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:3182)
	at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2841)
	at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:122)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:1605)
	at org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:954)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:679)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.getPropertiesForGenerator(RDBMSStoreManager.java:2045)
	at org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:1365)
	at org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl.java:3827)
	at org.datanucleus.state.JDOStateManager.setIdentity(JDOStateManager.java:2571)
	at org.datanucleus.state.JDOStateManager.initialiseForPersistentNew(JDOStateManager.java:513)
	at org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(ObjectProviderFactoryImpl.java:232)
	at org.datanucleus.ExecutionContextImpl.newObjectProviderForPersistentNew(ExecutionContextImpl.java:1414)
	at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2218)
	at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065)
	at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913)
	at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
	at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)
	... 58 more

We're using them in our testdependencies. https://github.com/apache/incubator-iceberg/blob/master/versions.props#L19-L20
It looks like for Spark, they've disabled the tests:
cenyuhai/spark@58cc0df

Also Spark 2.x Hive is on Datanucleaus 3.x: https://mvnrepository.com/artifact/org.apache.spark/spark-hive_2.12/2.4.4

We could run an external hive metastore instead of an embedded one? I have PoC here: https://github.com/godatadriven-dockerhub/hive-metastore#run-it-local-using-iceberg-incubating

rdblue · 2019-11-06T19:05:55Z

@Fokko, that project to bring up a metastore in Docker is great! We should point people to it from the Iceberg docs for people to be able to test tables.

I'm not sure that a docker container would work for tests, though. Requiring a docker install to run tests is difficult. Is it possible to update Datanucleus and still have the 1.2.1 HMS work?

rdblue · 2020-02-17T00:15:34Z

We just had a discussion about Hive metastore versions on the dev list. What I suggested was updating to the version that Spark uses when Spark 3.0 is out. That should pull in a much newer version and hopefully solve these issues. Does that sound reasonable?

Fokko · 2020-02-17T08:55:28Z

Thanks for the update. I've just subscribed to the devlist as well, just to keep in the loop :)

Related, updating Avro in Hive is a bit of the chicken-egg problem, more background here: https://jira.apache.org/jira/browse/HIVE-21737 (at the end)

That definitely sounds reasonable.

rdblue · 2020-02-20T18:01:25Z

If we update to Hive 2.3, does that fix tests?

rdblue · 2020-04-04T21:12:57Z

Looks like even with the new Hive version, Java 11 is still broken.

Fokko · 2020-04-11T05:53:17Z

Looks like an internal Hive issue to me:

MetaException(message:Got exception: java.lang.ClassCastException class [Ljava.lang.Object; cannot be cast to class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module java.base of loader 'bootstrap'))
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1390)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:204)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:129)
	at org.apache.iceberg.hive.HiveMetastoreTest.startMetastore(HiveMetastoreTest.java:43)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.base/java.lang.Thread.run(Thread.java:834)

Fokko · 2020-04-11T05:54:05Z

Spark also has the same issue: https://issues.apache.org/jira/browse/SPARK-29245

Will be solved in Hive 2.3.7 as I understand it: https://www.mail-archive.com/dev@hive.apache.org/msg138085.html

RC should be there soon: http://mail-archives.apache.org/mod_mbox/hive-dev/202004.mbox/browser

massdosage · 2020-04-11T09:16:32Z

I have been working with the Hive community to get a Hive 2.3.7 release out which should resolve this issue and allow the usage of Hive libraries on JDK >=9. A release candidate has been created (you can test it out if you want - https://repository.apache.org/content/repositories/orgapachehive-1100/) and here is the discussion on the Hive dev list where a release has been proposed - http://mail-archives.apache.org/mod_mbox/hive-dev/202004.mbox/%3CCANQiJeXh9hVma%3DA88vPLnWtguDfw%2BrZ4%2BoqeoYTMsfDZyxQNgg%40mail.gmail.com%3E. As soon as that is released I would recommend Iceberg change its default Hive version to 2.3.7 instead of 2.3.6.

Fokko · 2020-04-13T08:56:41Z

Thanks for the update @massdosage. I'll keep an eye out on the Hive RC's so I can test those here as well.

massdosage · 2020-04-20T13:42:33Z

Hive 2.3.7 was released over the weekend but I think you already noticed! :)

Fokko · 2020-04-20T13:43:23Z

@massdosage Thank you. I've just updated the PR :) I wanted to run against the RC as well, but the release got through so quickly.

rdblue · 2020-04-21T16:40:44Z

I added better logging for test failures in #945. I'm going to close and re-open this to trigger a new build.

Fokko · 2020-04-21T16:54:44Z

Forgot to update the test dependencies, should be fixed now! :)

rdblue · 2020-04-21T18:06:24Z

Looks like Spark 2.4 doesn't support Java 11, according to this thread about ASM 6/7.

I think that's the current problem. Maybe we can try overriding ASM 6 with ASM 7, but we might just have to wait for a version of Spark that can work with Java 11.

Fokko · 2020-04-21T18:26:43Z

Yes, looks like it: apache/spark@3ed91c9

We can't force it to ASM7, since it part of the package path: import org.apache.xbean.asm7._

rdblue · 2020-04-21T18:32:33Z

Does it work to set the JVM target to Java 8 and still run in Java 11?

Fokko · 2020-04-21T20:00:07Z

I think this is already the case: https://github.com/apache/incubator-iceberg/blob/master/build.gradle#L82-L83

…into add-java-11

rdblue · 2020-07-01T20:43:30Z

I think this is finally unblocked. We now have Spark 2 and Spark 3 modules in master, so we could add tests with Java 11 that disable the Spark 2 modules.

rdblue · 2020-07-05T20:51:29Z

@Fokko, I had a chance to look at this yesterday and it works if we disable the Spark 2 modules when running the build with JDK 11. I opened #1168 with the changes.

Fokko · 2020-07-06T16:39:11Z

Thanks for picking this up @rdblue I was stuck in other projects :)

Fokko mentioned this pull request Oct 25, 2019

Make Iceberg work with Java11 #578

Closed

Fokko force-pushed the add-java-11 branch from b294857 to ee6c0d1 Compare November 3, 2019 19:01

Fokko mentioned this pull request Nov 3, 2019

Add Java 11 support #602

Closed

rdblue mentioned this pull request Mar 19, 2020

Allow iceberg build with Java 8 or above version #850

Closed

rdblue mentioned this pull request Apr 20, 2020

Document Java version should be exact 1.8 #938

Merged

rdblue closed this Apr 21, 2020

rdblue reopened this Apr 21, 2020

Fokko force-pushed the add-java-11 branch from 617a414 to 1e80387 Compare April 21, 2020 19:55

Bump to Java 11

341c14c

Fokko force-pushed the add-java-11 branch from 1e80387 to 341c14c Compare April 21, 2020 19:57

Fokko added 2 commits April 21, 2020 22:10

Try to update to Spark 3

edc0f1c

Merge branch 'master' of https://github.com/apache/incubator-iceberg …

cd6959c

…into add-java-11

massdosage mentioned this pull request Jun 1, 2020

Upgrade Hive from 2.3.6 to 2.3.7 #1084

Merged

rdblue mentioned this pull request Jul 6, 2020

Add JDK 11 to CI build, disable Spark 2.4 with JDK 11 #1168

Merged

rdblue closed this in #1168 Jul 6, 2020

Fokko deleted the add-java-11 branch July 6, 2020 16:38

sunchao pushed a commit to sunchao/iceberg that referenced this pull request May 9, 2023

[Internal] Add AppleAwsClientFactory for Mascot (apache#577)

8b0e2c3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Java 11 to the testsuite #577

Add Java 11 to the testsuite #577

Fokko commented Oct 25, 2019

rdblue commented Oct 27, 2019

rdblue commented Nov 1, 2019

Fokko commented Nov 3, 2019 •

edited

rdblue commented Nov 6, 2019

rdblue commented Feb 17, 2020

Fokko commented Feb 17, 2020

rdblue commented Feb 20, 2020

rdblue commented Apr 4, 2020

Fokko commented Apr 11, 2020

Fokko commented Apr 11, 2020 •

edited

massdosage commented Apr 11, 2020

Fokko commented Apr 13, 2020

massdosage commented Apr 20, 2020 •

edited

Fokko commented Apr 20, 2020

rdblue commented Apr 21, 2020

Fokko commented Apr 21, 2020

rdblue commented Apr 21, 2020

Fokko commented Apr 21, 2020

rdblue commented Apr 21, 2020

Fokko commented Apr 21, 2020

rdblue commented Jul 1, 2020

rdblue commented Jul 5, 2020

Fokko commented Jul 6, 2020

Add Java 11 to the testsuite #577

Add Java 11 to the testsuite #577

Conversation

Fokko commented Oct 25, 2019

rdblue commented Oct 27, 2019

rdblue commented Nov 1, 2019

Fokko commented Nov 3, 2019 • edited

rdblue commented Nov 6, 2019

rdblue commented Feb 17, 2020

Fokko commented Feb 17, 2020

rdblue commented Feb 20, 2020

rdblue commented Apr 4, 2020

Fokko commented Apr 11, 2020

Fokko commented Apr 11, 2020 • edited

massdosage commented Apr 11, 2020

Fokko commented Apr 13, 2020

massdosage commented Apr 20, 2020 • edited

Fokko commented Apr 20, 2020

rdblue commented Apr 21, 2020

Fokko commented Apr 21, 2020

rdblue commented Apr 21, 2020

Fokko commented Apr 21, 2020

rdblue commented Apr 21, 2020

Fokko commented Apr 21, 2020

rdblue commented Jul 1, 2020

rdblue commented Jul 5, 2020

Fokko commented Jul 6, 2020

Fokko commented Nov 3, 2019 •

edited

Fokko commented Apr 11, 2020 •

edited

massdosage commented Apr 20, 2020 •

edited