Testing of MSSQL on Windows with AppVeyor #2617

photodude · 2017-01-26T16:46:41Z

This is an initial draft [POC] for testing of MSSQL on windows with AppVeyor

This PR creates the initial draft configuration files for AppVeyor CI testing, adjustments may be needed to meet project configuration needs.

For more information see issue #2616

photodude · 2017-01-26T16:47:37Z

Example build on AppVeyor https://ci.appveyor.com/project/photodude/dbal/build/1.0.17

/ping @deeky666

photodude · 2017-01-26T18:07:12Z

@deeky666 I'm not sure why the configuration xml files are failing to be read. I think that's all that needs to be sorted out (but we'll see once we can get tests running)

photodude · 2017-01-26T18:51:52Z

@deeky666 Based on what I have seen in some forums there might be something wrong in the Bootstrap file configuration listed in the xml files as bootstrap="../../vendor/autoload.php See this reference http://stackoverflow.com/questions/32274862/phpunit-could-not-read-phpunit-xml-on-travis-ci

deeky666 · 2017-01-26T21:00:21Z

@photodude first of all thanks for the amazing effort! Much appreciated! I guess the error might be a Windows related issue, because the exact same configuration is working on Linux. You might be right and the current working directory might be evaluated differently in this setup. Maybe you could try to play around with the path to the Composer autoloader. Maybe something like vendor/autoload.php or ./vendor/autoload.php (if maybe the cwd is evaluated to the calling context and not based on the phpunit configuration file location).

photodude · 2017-01-27T00:04:49Z

@deeky666 According to another forum post I read maybe the bootstrap file is doing something like the following

chdir(dirname(__FILE__) . "/../src/");

So that classes and autoload would find actual classes that are separated from the test classes.

see http://stackoverflow.com/questions/21599640/phpunit-needs-an-absolute-path-to-phpunit-xml

I do agree that it is some kind of windows specific configuration issue.

photodude · 2017-01-27T00:07:04Z

One thought is we could copy the bootstrap file to this repo and make appveyor specific modifications rather than letting it live in the vendor directory.

photodude · 2017-01-27T00:14:20Z

A note I found about the bootstrap file used in the XML located in another repo file

'vendor/autoload.php could not be found. Did you run `php composer.phar install`?'

Seems this file is related to composer???

photodude · 2017-01-27T00:37:31Z

ok Looking at the directory the \tests and \docs folders are not being imported from the repo. so literally the files are not there.

@deeky666 could this be a problem with the .gitattributes listing /tests export-ignore ?

photodude · 2017-01-27T00:43:23Z

Yep the root issue is the .gitattributes listing /tests export-ignore

Tests are now running https://ci.appveyor.com/project/photodude/dbal/build/1.0.21

Will review later to see if there are other configuration issues

photodude · 2017-01-27T01:03:49Z

I'm not sure why but the builds are failing due to

  [Seld\JsonLint\ParsingException]                                        
  "./composer.lock" does not contain valid JSON                           
  Parse error on line 1:                                                  
  ^                                                                       
  Expected one of: 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['

https://ci.appveyor.com/project/photodude/dbal/build/1.0.24/job/b3ur560vhh14xg62#L112

Seems like composer suddenly decided to be mad

photodude · 2017-01-27T01:05:25Z

Maybe We should just kill the composer cache....

photodude · 2017-01-27T02:20:38Z

@deeky666 Looks like everything is running and passing now... minus the PGSQL tests which have one failure and some test is killing the overall unit testing progress (could use debug to find the last test which is stalling out)

https://ci.appveyor.com/project/photodude/dbal/build/1.0.25

deeky666 · 2017-01-27T10:41:15Z

.gitattributes

@@ -1,4 +1,3 @@
-/tests export-ignore


You have to revert this, otherwise we cannot provide lightweight packages anymore. I think the problem is shallow_clone in the CI config which seems to download the zip archive rather than really cloning the repository.

I've reverted the change and I'll fully remove shallow_clone to fix the underlying issue. Thanks for the tip.

This has been implemented

deeky666 · 2017-01-27T10:42:13Z

.appveyor.yml

+  - mssql2014
+  - mysql
+  - postgresql94
+  - iis


Is this required for SQL Server?

IIS is the webserver for windows (unless you install apache...something a bit more complicated)

deeky666 · 2017-01-27T10:43:35Z

.appveyor.yml

+  - SET COMPOSER_NO_INTERACTION=1
+  - SET PHP=1
+  - SET ANSICON=121x90 (121x90)
+services:


Can we make the services load only if the current build environment requires them? Like "load mysql only if %db == 'mysql'".

on travis we can put services in the matrix, I'm not sure if that is possible here.

Maybe like this?

I was just looking at that same item for these changes https://github.com/photodude/dbal/blob/patch-2/.appveyor.yml#L106-L113

This has been implemented

deeky666 · 2017-01-27T11:00:41Z

.appveyor.yml

+
+## Install PHP and composer, and run the appropriate composer command
+install:
+    - IF EXIST c:\tools\php (SET PHP=0)


Do we really need this? PHP has to be installed on each build anyways, doesn't it? Can we get rid of the PHP variable?

I'm following a guide on the setup of php on appveyor. I've also been wondering If it's necessary or superfluous. https://blog.wyrihaximus.net/2016/11/running-php-unit-tests-on-windows-using-appveyor-and-chocolatey/

Hmm weird it is not really explained there. Makes things quite complicated. We might leave it as is for now and improve later on.

deeky666 · 2017-01-27T11:05:43Z

tests/appveyor/mariadb.appveyor.xml

@@ -0,0 +1,37 @@
+<?xml version="1.0" encoding="utf-8"?>


You can remove this file, as we currently do not have MariaDB in the appveyor build matrix

It's a placeholder, I'm looking into the possibility of installing MariaDB in appveyor as there is a windows version.

I've got MariaDB installed and running, but some adjustment to the DSN user and password are needed to get those tests to pass

MariaDB is working now

deeky666 · 2017-01-27T11:09:48Z

tests/appveyor/mssql.appveyor.xml

+    <php>
+        <ini name="error_reporting" value="-1" />
+
+        <var name="db_type" value="pdo_sqlvr"/>


Can you change the names of the config files to the convention <dbal_driver_name>.appveyor.xml so that we can have a test config for each driver? Then also please can you add another build for sqlsrv driver?

I followed and copied the Travis files for the config XMLs, is there a list I can use to make sure I have the right dbal driver names for the change?

I believe we can also test on multiple MSSQL versions, if we want to wait for a longer list of builds to run. Also note, each version of MSSQL has a specific host name i.e. (local)\SQL2014 which is set in the xml files

You can look up the driver names in the Driver classes like here.
A complete list of supported drivers currently is:

ibm_db2

mysqli

oci8

pdo_mysql

pdo_oracle

pdo_pgsql

pdo_sqlite

pdo_sqlsrv

sqlanywhere

sqlsrv

Simply leave those out that cannot (yet) be run on AppVeyor.

If different database versions and or vendors (MySQL/MariaDB) require different connection parameters, we should probably extend the convention to something like (<db_name>(.<db_version>)?.)?<dbal_driver_name>.appveyor.xml

I'll have to think about how to implement that suggested change in a way that could work with - vendor\bin\phpunit --verbose --debug -c tests\appveyor\%db%.appveyor.xml
and with If ($env:db -like "mysql")

I'd suggest that you define multiple environment variables in your build matrix like:

db

db_version

driver

Then you can do something like phpunit -c tests\appveyor\%db%.%db_version%.%driver%.appveyor.xml

Someone should also make the same kinds of changes to the Travis config XMLs

Yes that could need some improvement, too.

Considering the config XML files live in a CI service specific option can we drop the service name from the XML files?
so change tests/appveyor/<db_name>.<db_version>.<dbal_driver_name>.appveyor.xml to tests/appveyor/<db_name>.<db_version>.<dbal_driver_name>.xml ?

deeky666 · 2017-01-27T11:10:57Z

tests/appveyor/mysql.appveyor.xml

+
+    <testsuites>
+        <testsuite name="Doctrine DBAL Test Suite">
+            <directory>C:\projects\dbal\tests\Doctrine\Tests\DBAL</directory>


Why is an absolute path required here? Can we use ../Doctrine/Tests/DBAL?

that was a change related to figuring out why the config files could not be read. I'll revert the related changes.

deeky666 · 2017-01-27T16:11:13Z

tests/appveyor/mssql.appveyor.xml

+    <php>
+        <ini name="error_reporting" value="-1" />
+
+        <var name="db_type" value="pdo_sqlvr"/>


You can look up the driver names in the Driver classes like here.
A complete list of supported drivers currently is:

ibm_db2

mysqli

oci8

pdo_mysql

pdo_oracle

pdo_pgsql

pdo_sqlite

pdo_sqlsrv

sqlanywhere

sqlsrv

Simply leave those out that cannot (yet) be run on AppVeyor.

If different database versions and or vendors (MySQL/MariaDB) require different connection parameters, we should probably extend the convention to something like (<db_name>(.<db_version>)?.)?<dbal_driver_name>.appveyor.xml

deeky666 · 2017-01-27T16:12:37Z

.appveyor.yml

+  - SET COMPOSER_NO_INTERACTION=1
+  - SET PHP=1
+  - SET ANSICON=121x90 (121x90)
+services:


Maybe like this?

deeky666 · 2017-01-27T16:13:21Z

.appveyor.yml

+
+## Install PHP and composer, and run the appropriate composer command
+install:
+    - IF EXIST c:\tools\php (SET PHP=0)


Hmm weird it is not really explained there. Makes things quite complicated. We might leave it as is for now and improve later on.

deeky666 · 2017-01-27T16:14:22Z

tests/appveyor/sqlsrv.appveyor.xml

+    <php>
+        <ini name="error_reporting" value="-1" />
+
+        <var name="db_type" value="sqlvr"/>


typo: sqlsrv

deeky666 · 2017-01-27T16:16:55Z

.appveyor.yml

+#  - mssql2014
+#  - mysql
+#  - postgresql94
+  - iis


Why is a webserver required at all in our test setup? Is it required for SQL Server to run?

I'll try without it

deeky666 · 2017-01-27T16:45:37Z

.appveyor.yml

+          echo extension=php_pdo_sqlsrv_nts.dll >> php.ini}
+        If ($env:PHP -eq "1") {
+          echo extension=php_sqlsrv_nts.dll >> php.ini}}
+    - IF %PHP%==1 echo extension=php_pdo_pgsql.dll >> php.ini


Maybe if you work with DBAL driver names as environment variable you could even substitute it here and save some lines. Like: IF %PHP%==1 echo extension=php_%driver%.dll >> php.ini

I'll leave that for future improvement considerations. Let's get the core functionality in place and working.

Ok. Agreed. :)

deeky666 · 2017-01-27T16:47:54Z

.appveyor.yml

+            If ($env:db -like "mssql") {
+              appveyor DownloadFile https://github.com/Microsoft/msphpsql/releases/download/4.1.5-Windows/7.0.zip
+              7z x 7.0.zip > $null
+              copy 7.0\x64\php_pdo_sqlsrv_7_nts.dll ext\php_pdo_sqlsrv_nts.dll


I would copy the DLLs to php_pdo_sqlsrv.dll instead of php_pdo_sqlsrv_nts.dll. This makes the naming consistent with the other driver's DLLs and you can potentially make use of variable substitution to save if...else stuf

I followed this convention since there is a TS (threadsafe) and a NTS (not threadsafe) versions for the drivers. chocolatey is currently installing the NTS version of PHP but are implementing a change that will allow forcing the TS version of PHP.

As for the condition, it is only in there to prevent copying the DLL line into the PHP.ini file when we don't have the DLLs downloaded (prevents warnings about being unable to load dll)

We don't care about thread safety, that is nothing to worry about for us. So I don't see an issue with only considering the NTS driver. But I have no strong opinion about this. Just thought it could save some lines later when using variable substitution.

might be worth considering rewriting this section with cinst wget -y and wget the dll's as a faster option

https://www.microsoft.com/en-us/sql-server/developer-get-started/php-windows

Looks like the DLLs will not register if they are not named with the NTS

Sadly wget was no faster and the PDO driver wouldn't always download.

deeky666 · 2017-01-27T16:50:26Z

.appveyor.yml

+    - IF %PHP%==1 echo @php %%~dp0composer.phar %%* > composer.bat
+    - appveyor-retry appveyor DownloadFile https://getcomposer.org/composer.phar
+    - cd c:\projects\dbal
+    - IF %dependencies%==lowest appveyor-retry composer update --prefer-lowest --no-progress --profile -n


I still don't get this dependencies stuff from composer. Do we need it? You have set the environment variable to fixed current anyways...

Is there ever a time when testing might include lowest or highest dependencies? I set to current for the POC just for quicker testing

I have never used it nor seen it at all. So I guess that kind of thing is not interesting for us. We do not have such thing in Travis either.

If seen it in a few places where it's used in the "allowed failures" to get an idea of what's coming next and if the next release of a dependency breaks a package.

There are also times where it's used to allow for testing when the consuming repos might have higher or lower requirements and the package itself is typically happy in any case with a broader range of dependency versions (put they want to test for it when changes are made).

I have removed the composer high/low dependencies stuff

photodude · 2017-01-27T18:07:36Z

@deeky666 after making the config XML file name changes I'm now seeing an issue where pdo_sqlsrv will fail as driver not found. https://ci.appveyor.com/project/photodude/dbal/build/1.0.50/job/cvckl39ntwh1gamo#L362

related xml line https://github.com/photodude/dbal/blob/patch-2/tests/appveyor/mssql.mssql2014.pdo_sqlsrv.appveyor.xml#L11

any thoughts?

deeky666 · 2017-01-27T18:31:22Z

Looks like the pdo_sqlsrv driver was not loaded. Can you check by php -m or else if it got loaded? Guess it has to do with your conditional if...else loading (not sure).

deeky666 · 2017-01-27T18:33:09Z

.appveyor.yml

+    driver: sqlsrv
+    db_version: mssql2014
+    php_ver_target: 7.0
+  - db: sqlsrv


this needs to be mssql

photodude · 2017-01-27T20:17:23Z

so the NTS part of the DLL is required, still trying to figure out why the PDO SQLSRV driver is not loading

photodude · 2017-01-31T16:43:24Z

Just need to figure out why we are getting the following errors in the build
https://ci.appveyor.com/project/photodude/dbal/build/1.0.87/job/aw7kdhdsudiv41s0#L304

PDOException: SQLSTATE[28000]: [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Login failed for user 'sa'.

and https://ci.appveyor.com/project/photodude/dbal/build/1.0.87/job/6j0lu02x4ao27i6s#L304

SQLSrvException: SQLSTATE [28000, 18456]: [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Login failed for user 'sa'.
SQLSTATE [42000, 4060]: [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Cannot open database "doctrine_tests" requested by the login. The login failed.
SQLSTATE [28000, 18456]: [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Login failed for user 'sa'.
SQLSTATE [42000, 4060]: [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Cannot open database "doctrine_tests" requested by the login. The login failed.

following the docs the DSN info should be correct here
https://www.appveyor.com/docs/services-databases/#sql-server-2014

photodude · 2017-01-31T16:56:07Z

Looking at the build times I suggest we consider just testing the MSSQL drivers on AppVeyor

(unless the Doctrine Project wants to consider paying for additional concurrent jobs so parallel testing could be implimented)

deeky666 · 2017-02-01T11:54:29Z

@photodude looks great so far. Concerning the build failures, I have two clues:

It's weird that failures are only related to Azure tests. So either the test setup is not working (those tests use a different base test class and setup) or there is a problem with Azure related server setup.
The failure indicates that the doctrine_tests DB is not available. This is quite expected as the database is not created by Doctrine AFAIK. What confuses me though is that all of the other functional tests do NOT fail even though I would expect them to in this situation.

Can you try to change this line to return true? I don't know why the platform is configured as not supporting creating and dropping databases because in theory it should be able to (maybe a bug). With that line changed, the testsuite should create the database automatically.

photodude · 2017-02-01T16:20:51Z

I'll try that change, if it's a bug see bc8a6df

photodude · 2017-02-01T16:37:50Z

@deeky666 Looks like changing that line to true made no difference.

I've got some suggested help from AppVeyor's @IlyaFinkelshteyn who said the databases are not being created. The suggestion is to check the part our script that is responsible for database creation. see support topic http://help.appveyor.com/discussions/problems/6010-appveyoryml-scripting-issues#comment_41862813

So we possibly have database creation issues, maybe the Sharding or Sharding tests for SQL are bugged as well

Majkl578

LGTM once squashed (i.e. one commit with AppVeyor changes + test fixes from @morozov, please drop my empty commit).

reworked

Majkl578

AppVeyor green

morozov

Looks like caching of Chocolatey and Composer packages could use some improvement, but it can be done later.

👍

Ocramius · 2018-03-16T10:08:15Z

This is awesome! @photodude thanks a lot for pushing this forward: it was a heck of a marathon!

Majkl578 · 2018-03-16T14:45:14Z

@morozov Note: While rebasing develop, this either needs to be left out or adjusted to --ignore-platform-reqs (afaik there is no 7.2-specific code yet) due the PHP 7.2 requirement.

@photodude Can you please link some issue about the 7.2 so we can keep an eye on it? I want to port this to ORM too and it requires 7.2 already.

photodude · 2018-03-16T23:39:27Z

@Majkl578 See #3056 for a php 7.2 fix

photodude · 2018-03-16T23:47:50Z

Thanks everyone for all the help getting this complete and merged in.
You're welcome @Ocramius Over a year in the making, I'm happy to see it merged.

Shout out to @deeky666, @morozov, @Majkl578 who got the tests into the green and made all the suggestions for getting this into a state to merge. Without their help this would still be a work in progress.

morozov · 2018-03-16T23:56:43Z

@photodude thank you for not giving this up! It's an incredible achievement for the project.

photodude · 2018-03-16T23:58:26Z

@Majkl578 you'll want to watch this issue appveyor/ci#2155
that issue is regarding the problem with the appveyor command-line utility to download resources.
Once fixed the powershell Invoke-WebRequest Workaround can be replaced with the simple single line download.
reference on the basics of Appveyor testing of php projects
https://blog.wyrihaximus.net/2016/11/running-php-unit-tests-on-windows-using-appveyor-and-chocolatey/

photodude · 2018-03-16T23:59:44Z

Thanks @morozov It's nice to see this implemented so there is better testing with more supported platforms included in the automatic tests.

deeky666 · 2018-03-19T09:03:38Z

great job @photodude thank you!

morozov · 2018-03-26T22:08:02Z

.appveyor.yml

+  - ps: >-
+      if ($env:db -eq "mssql") {
+        $instanceName = $env:db_version.ToUpper()
+        Start-Service "MSSQL`$$instanceName"


@photodude is there a reason we use Start-Service and not net start? As per the Appveyor's suggestion in appveyor/ci#2204 (comment), it could be the reason for intermittent login failures.

It's based on appveyor documentation https://www.appveyor.com/docs/services-databases/

If they have some new thing they suggest to do, their documentation does not reflect that.

@photodude thanks. Trying to change this in #3071.

…gin failures See doctrine#2617 (comment).

deeky666 added CI New Feature Testing labels Jan 26, 2017

deeky666 previously requested changes Jan 27, 2017

View reviewed changes

deeky666 reviewed Jan 27, 2017

View reviewed changes

DavidBadura mentioned this pull request Jan 31, 2017

add official MSSQL support sonata-project/EntityAuditBundle#260

Closed

Fixed fetching last inserted ID from a SQL Server connection

cadd79c

Majkl578 previously approved these changes Mar 16, 2018

View reviewed changes

morozov dismissed Majkl578’s stale review via cadd79c March 16, 2018 05:49

morozov force-pushed the patch-2 branch from ce41f5f to cadd79c Compare March 16, 2018 05:49

Majkl578 added SQL Server and removed WIP labels Mar 16, 2018

Majkl578 changed the title ~~[POC] testing of MSSQL on windows with AppVeyor~~ Testing of MSSQL on Windows with AppVeyor Mar 16, 2018

Majkl578 approved these changes Mar 16, 2018

View reviewed changes

morozov approved these changes Mar 16, 2018

View reviewed changes

Ocramius merged commit 9e68c3c into doctrine:master Mar 16, 2018

Ocramius mentioned this pull request Mar 16, 2018

[RFC] CI testing of MSSQL on windows with AppVeyor #2616

Closed

Majkl578 mentioned this pull request Mar 16, 2018

Improve handling of schemas in SQL Server >= 2008 #3020

Merged

photodude mentioned this pull request Mar 16, 2018

Set Appveyor to use PHP 7.2 #3056

Merged

photodude deleted the patch-2 branch March 16, 2018 23:39

Majkl578 mentioned this pull request Mar 17, 2018

ref #2597 Fix function getListTablesSQL() for SQL Server #2598

Closed

morozov reviewed Mar 26, 2018

View reviewed changes

morozov added a commit to morozov/dbal that referenced this pull request Mar 27, 2018

Start SQL Server using a synchronous command to avoid intermittent lo…

b0591db

…gin failures See doctrine#2617 (comment).

morozov mentioned this pull request Mar 27, 2018

Start SQL Server using a synchronous command to avoid intermittent login failures #3071

Merged

bertoverflow mentioned this pull request Sep 19, 2018

Incorrect Handling of LastInsertId in SQLSrvConnection #3293

Closed

morozov mentioned this pull request Jul 6, 2019

Invalid last inserted id when the table has a trigger with insert statements in SQLSrvConnection #3516

Open

github-actions bot locked as resolved and limited conversation to collaborators Aug 13, 2022

Testing of MSSQL on Windows with AppVeyor #2617

Testing of MSSQL on Windows with AppVeyor #2617

Conversation

photodude commented Jan 26, 2017

photodude commented Jan 26, 2017 • edited Loading

photodude commented Jan 26, 2017

photodude commented Jan 26, 2017

deeky666 commented Jan 26, 2017

photodude commented Jan 27, 2017

photodude commented Jan 27, 2017 • edited Loading

photodude commented Jan 27, 2017

photodude commented Jan 27, 2017 • edited Loading

photodude commented Jan 27, 2017 • edited Loading

photodude commented Jan 27, 2017

photodude commented Jan 27, 2017

photodude commented Jan 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

photodude Jan 31, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

photodude Jan 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

photodude Jan 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

photodude Feb 4, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

photodude commented Jan 27, 2017

deeky666 commented Jan 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

photodude commented Jan 27, 2017

photodude commented Jan 31, 2017 • edited Loading

photodude commented Jan 31, 2017

deeky666 commented Feb 1, 2017 • edited Loading

photodude commented Feb 1, 2017

photodude commented Feb 1, 2017 • edited Loading

Majkl578 left a comment

photodude commented Jan 26, 2017 •

edited

Loading

photodude commented Jan 27, 2017 •

edited

Loading

photodude commented Jan 27, 2017 •

edited

Loading

photodude commented Jan 27, 2017 •

edited

Loading

photodude commented Jan 27, 2017 •

edited

Loading

photodude Jan 31, 2017 •

edited

Loading

photodude Jan 27, 2017 •

edited

Loading

photodude Jan 27, 2017 •

edited

Loading

photodude Feb 4, 2017 •

edited

Loading

photodude commented Jan 31, 2017 •

edited

Loading

deeky666 commented Feb 1, 2017 •

edited

Loading

photodude commented Feb 1, 2017 •

edited

Loading

photodude commented Mar 16, 2018 •

edited by Ocramius

Loading

photodude commented Mar 16, 2018 •

edited

Loading