Ubuntu 16.04 + Neo4j v3.0.1 -- installation via apt-get pulls in Java 9 #7188

MysterAitch · 2016-05-20T14:19:09Z

Per a conversation via Slack, here is an issue describing my installation experience today .

Host OS: Windows 7
Virtualbox guest OS: Ubuntu 16.04 server (fresh standard installation + OpenSSH server + apt-get upgrade)
Neo4j version: v3.0.1 (via apt-get instructions at http://debian.neo4j.org/ )

In particular, note that the standard apt-get installation of neo4j is pulling in openjdk-9-jre and openjdk-9-jre-headless.

 Followed installation instructions at http://debian.neo4j.org/ 
        wget -O - https://debian.neo4j.org/neotechnology.gpg.key | sudo apt-key add -
        echo 'deb http://debian.neo4j.org/repo stable/' >/tmp/neo4j.list
        sudo mv /tmp/neo4j.list /etc/apt/sources.list.d
        sudo apt-get update
        sudo apt-get install neo4j      

            <user>@<hostname>:~$ sudo apt-get install neo4j
            Reading package lists... Done
            Building dependency tree
            Reading state information... Done
            The following additional packages will be installed:
              at-spi2-core ca-certificates-java daemon dbus-x11 fontconfig fontconfig-config fonts-dejavu-core fonts-dejavu-extra gconf-service
              gconf-service-backend gconf2 gconf2-common hicolor-icon-theme java-common libasound2 libasound2-data libatk-bridge2.0-0
              libatk-wrapper-java libatk-wrapper-java-jni libatk1.0-0 libatk1.0-data libatspi2.0-0 libavahi-client3 libavahi-common-data
              libavahi-common3 libavahi-glib1 libbonobo2-0 libbonobo2-common libcairo2 libcanberra0 libcups2 libdatrie1 libdrm-amdgpu1
              libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libfontconfig1 libfontenc1 libgconf-2-4 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common
              libgif7 libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgnome-2-0 libgnome2-common libgnomevfs2-0 libgnomevfs2-common
              libgraphite2-3 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice6 libjbig0 libjpeg-turbo8 libjpeg8 liblcms2-2
              libllvm3.8 libltdl7 libnspr4 libnss3 libnss3-nssdb libogg0 liborbit-2-0 libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0
              libpciaccess0 libpcsclite1 libpixman-1-0 libsm6 libtdb1 libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libvorbis0a libvorbisfile3
              libx11-xcb1 libxaw7 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0 libxcb-shape0 libxcb-shm0 libxcb-sync1
              libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxft2 libxi6 libxinerama1 libxmu6 libxpm4 libxrandr2 libxrender1 libxshmfence1
              libxt6 libxtst6 libxv1 libxxf86dga1 libxxf86vm1 openjdk-9-jre openjdk-9-jre-headless sound-theme-freedesktop x11-common x11-utils
            Suggested packages:
              gconf-defaults-service default-jre libasound2-plugins alsa-utils libbonobo2-bin libcanberra-gtk0 libcanberra-pulse cups-common
              desktop-base libgnomevfs2-bin libgnomevfs2-extra gamin | fam gnome-mime-data librsvg2-common gvfs liblcms2-utils pcscd
              icedtea-8-plugin libnss-mdns fonts-ipafont-gothic fonts-ipafont-mincho ttf-wqy-microhei | ttf-wqy-zenhei fonts-indic mesa-utils
            The following NEW packages will be installed:
              at-spi2-core ca-certificates-java daemon dbus-x11 fontconfig fontconfig-config fonts-dejavu-core fonts-dejavu-extra gconf-service
              gconf-service-backend gconf2 gconf2-common hicolor-icon-theme java-common libasound2 libasound2-data libatk-bridge2.0-0
              libatk-wrapper-java libatk-wrapper-java-jni libatk1.0-0 libatk1.0-data libatspi2.0-0 libavahi-client3 libavahi-common-data
              libavahi-common3 libavahi-glib1 libbonobo2-0 libbonobo2-common libcairo2 libcanberra0 libcups2 libdatrie1 libdrm-amdgpu1
              libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libfontconfig1 libfontenc1 libgconf-2-4 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common
              libgif7 libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgnome-2-0 libgnome2-common libgnomevfs2-0 libgnomevfs2-common
              libgraphite2-3 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice6 libjbig0 libjpeg-turbo8 libjpeg8 liblcms2-2
              libllvm3.8 libltdl7 libnspr4 libnss3 libnss3-nssdb libogg0 liborbit-2-0 libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0
              libpciaccess0 libpcsclite1 libpixman-1-0 libsm6 libtdb1 libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libvorbis0a libvorbisfile3
              libx11-xcb1 libxaw7 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0 libxcb-shape0 libxcb-shm0 libxcb-sync1
              libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxft2 libxi6 libxinerama1 libxmu6 libxpm4 libxrandr2 libxrender1 libxshmfence1
              libxt6 libxtst6 libxv1 libxxf86dga1 libxxf86vm1 neo4j openjdk-9-jre openjdk-9-jre-headless sound-theme-freedesktop x11-common
              x11-utils
            0 upgraded, 113 newly installed, 0 to remove and 3 not upgraded.
            Need to get 272 MB of archives.
            After this operation, 577 MB of additional disk space will be used.
            Do you want to continue? [Y/n] y


            [...]
            Setting up neo4j (3.0.1) ...
            Adding system user `neo4j' (UID 111) ...
            Adding new user `neo4j' (UID 111) with group `nogroup' ...
            Not creating home directory `/var/lib/neo4j'.

This was discovered because the neo4j service would not start, per this:

/var/log/neo4j/neo4j.log:
Error: VM option 'hashCode' is experimental and must be enabled via -XX:+UnlockExperimentalVMOptions.
Error: The unlock option must precede 'hashCode'.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

<user>@<hostname>:~$ sudo service neo4j start
Job for neo4j.service failed because the control process exited with error code. See "systemctl status neo4j.service" and "journalctl -xe" for details.
<user>@<hostname>:~$ sudo service neo4j status
● neo4j.service - LSB: Neo4j Graph Database server
   Loaded: loaded (/etc/init.d/neo4j; bad; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2016-05-20 14:17:15 BST; 2s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1599 ExecStart=/etc/init.d/neo4j start (code=exited, status=1/FAILURE)
    Tasks: 0 (limit: 512)
   Memory: 22.3M
      CPU: 504ms

May 20 14:17:14 <hostname> systemd[1]: Starting LSB: Neo4j Graph Database server...
May 20 14:17:15 <hostname> neo4j[1599]: Starting Neo4j.
May 20 14:17:15 <hostname> neo4j[1599]: WARNING: Max 1024 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
May 20 14:17:15 <hostname> neo4j[1599]: /usr/share/neo4j/bin/neo4j: line 149: /var/run/neo4j/neo4j.pid: No such file or directory
May 20 14:17:15 <hostname> systemd[1]: neo4j.service: Control process exited, code=exited status=1
May 20 14:17:15 <hostname> systemd[1]: Failed to start LSB: Neo4j Graph Database server.
May 20 14:17:15 <hostname> systemd[1]: neo4j.service: Unit entered failed state.
May 20 14:17:15 <hostname> systemd[1]: neo4j.service: Failed with result 'exit-code'.
<user>@<hostname>:~$

spacecowboy · 2016-05-20T14:34:07Z

Something is very odd about this. Java9 is not released and should only exist in experimental sources.

Could you post the output for source in $(find /etc/apt -name "*.list"); cat $source please?

MysterAitch · 2016-05-20T14:44:35Z

@spacecowboy - Running that in the terminal doesn't work for me (perhaps I'm running it incorrectly?):

~$ for source in $(find /etc/apt -name "*.list"); cat $source
-bash: syntax error near unexpected token `cat'

I have only a single .list file in this /etc/apt/ and only neo4j within /etc/apt/sources.list.d/; is this the information you're looking for?

~$ find /etc/apt -name "*.list"
/etc/apt/sources.list.d/neo4j.list
/etc/apt/sources.list

~$ ls /etc/apt/
apt.conf.d  preferences.d  sources.list  sources.list~  sources.list.d  trusted.gpg  trusted.gpg~  trusted.gpg.d

~$ cat /etc/apt/sources.list | grep deb
# deb cdrom:[Ubuntu-Server 16.04 LTS _Xenial Xerus_ - Release amd64 (20160420.3)]/ xenial main restricted
#deb cdrom:[Ubuntu-Server 16.04 LTS _Xenial Xerus_ - Release amd64 (20160420.3)]/ xenial main restricted
deb http://gb.archive.ubuntu.com/ubuntu/ xenial main restricted
# deb-src http://gb.archive.ubuntu.com/ubuntu/ xenial main restricted
deb http://gb.archive.ubuntu.com/ubuntu/ xenial-updates main restricted
# deb-src http://gb.archive.ubuntu.com/ubuntu/ xenial-updates main restricted
deb http://gb.archive.ubuntu.com/ubuntu/ xenial universe
# deb-src http://gb.archive.ubuntu.com/ubuntu/ xenial universe
deb http://gb.archive.ubuntu.com/ubuntu/ xenial-updates universe
# deb-src http://gb.archive.ubuntu.com/ubuntu/ xenial-updates universe
deb http://gb.archive.ubuntu.com/ubuntu/ xenial multiverse
# deb-src http://gb.archive.ubuntu.com/ubuntu/ xenial multiverse
deb http://gb.archive.ubuntu.com/ubuntu/ xenial-updates multiverse
# deb-src http://gb.archive.ubuntu.com/ubuntu/ xenial-updates multiverse
deb http://gb.archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse
# deb-src http://gb.archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse
# deb http://archive.canonical.com/ubuntu xenial partner
# deb-src http://archive.canonical.com/ubuntu xenial partner
deb http://security.ubuntu.com/ubuntu xenial-security main restricted
# deb-src http://security.ubuntu.com/ubuntu xenial-security main restricted
deb http://security.ubuntu.com/ubuntu xenial-security universe
# deb-src http://security.ubuntu.com/ubuntu xenial-security universe
deb http://security.ubuntu.com/ubuntu xenial-security multiverse
# deb-src http://security.ubuntu.com/ubuntu xenial-security multiverse

~$ ls /etc/apt/sources.list.d/
neo4j.list

~$ cat /etc/apt/sources.list.d/neo4j.list | grep deb
deb http://debian.neo4j.org/repo stable/

spacecowboy · 2016-05-20T15:02:47Z

yes, but you might also have more .list files inside of /etc/apt/sources.list.d

What I was trying to ascertain was whether you something strange there from where java9 might come from.

Some googling around seems to indicate that Xenial Xerus actually ships openjdk-9 available by default however, so no error on your part.

Thanks for the report, we'll sort this out.. Meanwhile, it should be fine if you split it into two steps:

apt-get install openjdk-8
apt-get install neo4j

MysterAitch · 2016-05-20T15:08:52Z

I can confirm that installing default-jdk first (therefore openjdk8 by proxy) on a fresh installation of Ubuntu 16.04 does stop the installation of neo4j pulling in jdk9:

    Installation of JDK first (nb: note that default-jdk pulls in openjdk-8-jre openjdk-8-jre-headless)

        ~$ sudo apt-get install default-jdk
        Reading package lists... Done
        Building dependency tree
        Reading state information... Done
        The following additional packages will be installed:
          ca-certificates-java dbus-x11 default-jdk-headless default-jre default-jre-headless
          fontconfig fontconfig-config fonts-dejavu-core fonts-dejavu-extra gconf-service
          gconf-service-backend gconf2 gconf2-common hicolor-icon-theme java-common libasound2
          libasound2-data libasyncns0 libatk1.0-0 libatk1.0-data libavahi-client3
          libavahi-common-data libavahi-common3 libavahi-glib1 libbonobo2-0 libbonobo2-common
          libcairo2 libcanberra0 libcups2 libdatrie1 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2
          libdrm-radeon1 libflac8 libfontconfig1 libgconf-2-4 libgdk-pixbuf2.0-0
          libgdk-pixbuf2.0-common libgif7 libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgnome-2-0
          libgnome2-common libgnomevfs2-0 libgnomevfs2-common libgraphite2-3 libgtk2.0-0
          libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice-dev libice6 libjbig0 libjpeg-turbo8
          libjpeg8 liblcms2-2 libllvm3.8 libltdl7 libnspr4 libnss3 libnss3-nssdb libogg0 liborbit-2-0
          libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libpciaccess0 libpcsclite1
          libpixman-1-0 libpthread-stubs0-dev libpulse0 libsm-dev libsm6 libsndfile1 libtdb1
          libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libvorbis0a libvorbisenc2 libvorbisfile3
          libx11-dev libx11-doc libx11-xcb1 libxau-dev libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0
          libxcb-present0 libxcb-render0 libxcb-shm0 libxcb-sync1 libxcb1-dev libxcomposite1
          libxcursor1 libxdamage1 libxdmcp-dev libxfixes3 libxi6 libxinerama1 libxrandr2 libxrender1
          libxshmfence1 libxt-dev libxt6 libxtst6 libxxf86vm1 openjdk-8-jdk openjdk-8-jdk-headless
          openjdk-8-jre openjdk-8-jre-headless sound-theme-freedesktop x11-common x11proto-core-dev
          x11proto-input-dev x11proto-kb-dev xorg-sgml-doctools xtrans-dev
        Suggested packages:
          default-java-plugin gconf-defaults-service libasound2-plugins alsa-utils libbonobo2-bin
          libcanberra-gtk0 libcanberra-pulse cups-common desktop-base libgnomevfs2-bin
          libgnomevfs2-extra gamin | fam gnome-mime-data librsvg2-common gvfs libice-doc
          liblcms2-utils pcscd pulseaudio libsm-doc libxcb-doc libxt-doc openjdk-8-demo
          openjdk-8-source visualvm icedtea-8-plugin openjdk-8-jre-jamvm libnss-mdns
          fonts-ipafont-gothic fonts-ipafont-mincho ttf-wqy-microhei | ttf-wqy-zenhei fonts-indic
        The following NEW packages will be installed:
          ca-certificates-java dbus-x11 default-jdk default-jdk-headless default-jre
          default-jre-headless fontconfig fontconfig-config fonts-dejavu-core fonts-dejavu-extra
          gconf-service gconf-service-backend gconf2 gconf2-common hicolor-icon-theme java-common
          libasound2 libasound2-data libasyncns0 libatk1.0-0 libatk1.0-data libavahi-client3
          libavahi-common-data libavahi-common3 libavahi-glib1 libbonobo2-0 libbonobo2-common
          libcairo2 libcanberra0 libcups2 libdatrie1 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2
          libdrm-radeon1 libflac8 libfontconfig1 libgconf-2-4 libgdk-pixbuf2.0-0
          libgdk-pixbuf2.0-common libgif7 libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgnome-2-0
          libgnome2-common libgnomevfs2-0 libgnomevfs2-common libgraphite2-3 libgtk2.0-0
          libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice-dev libice6 libjbig0 libjpeg-turbo8
          libjpeg8 liblcms2-2 libllvm3.8 libltdl7 libnspr4 libnss3 libnss3-nssdb libogg0 liborbit-2-0
          libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libpciaccess0 libpcsclite1
          libpixman-1-0 libpthread-stubs0-dev libpulse0 libsm-dev libsm6 libsndfile1 libtdb1
          libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libvorbis0a libvorbisenc2 libvorbisfile3
          libx11-dev libx11-doc libx11-xcb1 libxau-dev libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0
          libxcb-present0 libxcb-render0 libxcb-shm0 libxcb-sync1 libxcb1-dev libxcomposite1
          libxcursor1 libxdamage1 libxdmcp-dev libxfixes3 libxi6 libxinerama1 libxrandr2 libxrender1
          libxshmfence1 libxt-dev libxt6 libxtst6 libxxf86vm1 openjdk-8-jdk openjdk-8-jdk-headless
          openjdk-8-jre openjdk-8-jre-headless sound-theme-freedesktop x11-common x11proto-core-dev
          x11proto-input-dev x11proto-kb-dev xorg-sgml-doctools xtrans-dev
        0 upgraded, 122 newly installed, 0 to remove and 3 not upgraded.
        Need to get 65.9 MB of archives.
        After this operation, 358 MB of additional disk space will be used.
        Do you want to continue? [Y/n] y


    Installation of neo4j (note very few dependencies)
        ~$ sudo apt-get install neo4j
        Reading package lists... Done
        Building dependency tree
        Reading state information... Done
        The following additional packages will be installed:
          daemon
        The following NEW packages will be installed:
          daemon neo4j
        0 upgraded, 2 newly installed, 0 to remove and 3 not upgraded.
        Need to get 61.3 MB of archives.
        After this operation, 69.9 MB of additional disk space will be used.
        Do you want to continue? [Y/n] y

spacecowboy · 2016-05-20T15:11:22Z

This appears to be a bug in Ubuntu, see this for example:
https://tickets.puppetlabs.com/browse/EZ-77

Our debian package explicitly requires java8-runtime so it shouldn't pull in java9

spacecowboy · 2016-05-20T15:43:41Z

Also reported to Ubuntu now:
https://bugs.launchpad.net/ubuntu/+source/openjdk-9/+bug/1584118


        Work around for OpenJDK 9 being in the Ubuntu 16.04 repos

The Ubuntu 16.04 repos include Java 9, despite its not having been released yet (see https://bugs.launchpad.net/ubuntu/+source/openjdk-9/+bug/1584118). Because the Java 9 package provides java8-runtime, this results in Java 9 being installed by default when people install our package on Ubuntu 16.04. This causes problems because: * We are not compatible with Java 9 because there is a change in command line argument order and the JVM fails to start. * We don't support Java 9 and have not tested against it. This change adds makes our package conflict with Java 9, so Java 8 is installed instead. Fixes neo4j#7188.


        Work around for OpenJDK 9 being in the Ubuntu 16.04 repos

The Ubuntu 16.04 repos include Java 9, despite its not having been released yet (see https://bugs.launchpad.net/ubuntu/+source/openjdk-9/+bug/1584118). Because the Java 9 package provides java8-runtime, this results in Java 9 being installed by default when people install our package on Ubuntu 16.04. This causes problems because: * We are not compatible with Java 9 because there is a change in command line argument order and the JVM fails to start. * We don't support Java 9 and have not tested against it. This change adds makes our package conflict with Java 9, so Java 8 is installed instead. Fixes neo4j#7188.

benbc · 2016-05-26T12:00:56Z

@MysterAitch The fix for this will be in Neo4j 3.0.2.

mal · 2016-08-12T11:50:49Z

I'm seeing the same issue with the 2.2.x releases, will this workaround be backported?

mal · 2016-08-12T12:01:08Z

Also the Ubuntu report appears to have been updated recently suggesting a different approach to fixing this issue than the one currently queued for release.

spacecowboy · 2016-08-12T13:14:13Z

@mal thanks for the notice. I have commented on the ubuntu issue why that doesn't work.

mal · 2016-08-12T13:18:33Z

FYI for anyone else experiencing this in the interim, it's possible to work around the issue like this:

Tested for neo4j (2.2.7, 2.2.10) on ubuntu 16.04

# cat /etc/apt/preferences.d/java 
Package: openjdk-9-*
Pin: release n=xenial
Pin-Priority: -10

spacecowboy · 2016-08-12T13:19:36Z

Another workaround is to manually install java8 before neo4j as I mention above, although your solution is a system wide solution which also fixes the issue for other packages

mal · 2016-08-12T13:56:06Z

Aha, you're quite correct. I arrived at the pinning solution after trying to install openjdk-8-jre-headless (because why install the whole jdk if it's not needed) before neo4j and getting nowhere fast. Turns out that it's not quite enough and that at a minimum it's openjdk-8-jre that must be installed prior (at least in the case 2.2.x).

spacecowboy · 2016-08-12T14:08:10Z

yes we have an independent issue that we depend on java-8-runtime instead of what is the correct one: java-8-runtime-headless

spacecowboy · 2016-08-16T16:10:34Z

@mal we have backported this to 2.2 and 2.3 now.

mal · 2016-08-16T16:50:04Z

Excellent! Thanks @spacecowboy, very much appreciated 😀

Do you happen to know when they'll make their way into http://debian.neo4j.org/repo stable/?

spacecowboy · 2016-08-16T21:46:46Z

@mal next patch release I'm afraid :(

ivastly · 2016-11-17T00:36:24Z

all temporary solutions mentioned here do not work for me (ubuntu 12.04).
apt-get install neo4j always requires java9 and breaks ((

benbc · 2016-11-17T08:55:02Z

@biganfa What version of Neo4j are you installing?

spacecowboy · 2016-11-17T09:08:14Z

@biganfa that's quite surprising since ubuntu 12.04 doesn't have java9 in its default repos. What extra repos/ppas have you enabled?

ivastly · 2016-11-17T09:57:49Z

@benbc @spacecowboy i did not do anything special, just typed apt-get install neo4j without any customization and it suggests to install java9. All advices in current thread (install java8 previously, edit some config file) did not help. Maybe i had some ppas in my system, i am not sure about that. Finally, i installed 16.04 on another box and neo4j 3.0.7 got running there

karpatit · 2018-01-16T06:06:10Z

Hello, I have many java version installed on my ubuntu system, as different applications require different java versions. I solved the issue by adding the following line at the beginning of /etc/neo4j/neo4j.conf:

JAVA_HOME=/usr/lib/jvm/java-8-oracle

After this it worked and I was able to connect to the web service...

spacecowboy added 3.0 operability labels May 20, 2016

spacecowboy added the bug label May 20, 2016

benbc mentioned this issue May 25, 2016

Work around for OpenJDK 9 being in the Ubuntu 16.04 repos #7222

Merged

benbc self-assigned this May 25, 2016

benbc closed this in 874c8da May 26, 2016

spacecowboy added the changelog label Jun 15, 2016

spacecowboy mentioned this issue Aug 15, 2016

Update debian java dependency #7728

Merged

8 of 8 tasks complete

neo4j/neo4j

Join GitHub today

Ubuntu 16.04 + Neo4j v3.0.1 -- installation via apt-get pulls in Java 9 #7188

Comments

MysterAitch commented May 20, 2016 • edited

This comment has been minimized.

spacecowboy commented May 20, 2016

This comment has been minimized.

MysterAitch commented May 20, 2016 • edited

This comment has been minimized.

spacecowboy commented May 20, 2016

This comment has been minimized.

MysterAitch commented May 20, 2016 • edited

This comment has been minimized.

spacecowboy commented May 20, 2016

This comment has been minimized.

spacecowboy commented May 20, 2016

This comment has been minimized.

benbc commented May 26, 2016

This comment has been minimized.

mal commented Aug 12, 2016

This comment has been minimized.

mal commented Aug 12, 2016

This comment has been minimized.

spacecowboy commented Aug 12, 2016

This comment has been minimized.

mal commented Aug 12, 2016

This comment has been minimized.

spacecowboy commented Aug 12, 2016 • edited

This comment has been minimized.

mal commented Aug 12, 2016

This comment has been minimized.

spacecowboy commented Aug 12, 2016

This comment has been minimized.

spacecowboy commented Aug 16, 2016

This comment has been minimized.

mal commented Aug 16, 2016

This comment has been minimized.

spacecowboy commented Aug 16, 2016

This comment has been minimized.

ivastly commented Nov 17, 2016

This comment has been minimized.

benbc commented Nov 17, 2016

This comment has been minimized.

spacecowboy commented Nov 17, 2016

This comment has been minimized.

ivastly commented Nov 17, 2016 • edited

This comment has been minimized.

karpatit commented Jan 16, 2018

MysterAitch commented May 20, 2016 •

edited

MysterAitch commented May 20, 2016 •

edited

MysterAitch commented May 20, 2016 •

edited

spacecowboy commented Aug 12, 2016 •

edited

ivastly commented Nov 17, 2016 •

edited