CMake: Support for DarwinSSL and mbedTLS + cleanup #1228

ligfx · 2017-01-26T00:45:10Z

No description provided.

mention-bot · 2017-01-26T00:45:12Z

@ligfx, thanks for your PR! By analyzing the history of the files in this pull request, we identified @Lekensteyn, @jzakrzewski and @jgsogo to be potential reviewers.

Lekensteyn · 2017-02-01T18:25:04Z

CMake/FindMbedTLS.cmake

+find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+
+set(MBEDTLS_INCLUDE_DIRS ${MBEDTLS_INCLUDE_DIR})
+set(MBEDTLS_LIBRARIES ${MBEDTLS_LIBRARY} ${MBEDX509_LIBRARY} ${MBEDCRYPTO_LIBRARY})


Probably a good idea to quote each of these variables (in case it contains spaces). E.g. "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" ...) (same for previous include_dirs line)

Ah, good catch! The quoting rules always trip me up.

Lekensteyn · 2017-02-01T18:31:19Z

CMakeLists.txt

+    if(SECURITY_FRAMEWORK)
+      set(SSL_ENABLED ON)
+      set(USE_DARWINSSL ON)
+      list(APPEND CURL_LIBS "-framework CoreFoundation -framework Security")


Does this work? Don't you have separate the two options like list(APPEND CURL_LIBS "-framework CoreFoundation" "-framework Security")?

(Specifying -framework should work according to cmake-commands manual (target_link_libraries), hopefully it does (untested).)

Any library that begins with - gets passed straight through to the command-line, so it works as two options or as one option. After taking another look at this code, I replaced this with "${COREFOUNDATION_FRAMEWORK}" "${SECURITY_FRAMEWORK}", since it seems more correct (and it already goes to the trouble of finding Security.framework, so might as well...)

webmaster128

Nice, thanks for pushing macOS+cmake. I'll test this on a Mac soon. The current change does not break my custom OpenSSL configuration on Linux, so 👍

webmaster128 · 2017-02-19T22:31:26Z

CMakeLists.txt


-if(CMAKE_USE_OPENSSL)
+option(CMAKE_USE_OPENSSL "Use OpenSSL code. Experimental" ON)
+if(CMAKE_USE_OPENSSL AND NOT SSL_ENABLED)


~~This silently prioritizes Secure Channel over OpenSSL when configuring CMAKE_USE_WINSSL=1 CMAKE_USE_OPENSSL=1 on windows.~~

~~I think this priority should at least be written down as a comment about TLS at the top, But the safe thing would be to disallow setting multiple TLS providers.~~

webmaster128 · 2017-02-20T11:02:52Z

When I apply this patch, I get a runtime error when setting CURLOPT_SSL_OPTIONS to CURLSSLOPT_NO_REVOKE. Any idea where this is coming from? It works with cmake build + OpenSSL and buildconf/configure build + DarwnSSL.

bagder · 2017-02-20T13:18:05Z

I get a runtime error

Can you elaborate on exactly what you get when you do what?

webmaster128 · 2017-02-20T13:41:52Z

The call of curl_easy_setopt with key CURLOPT_SSL_OPTIONS and value CURLSSLOPT_NO_REVOKE returns the code 48 = CURLE_UNKNOWN_OPTION.

bagder · 2017-02-20T14:01:25Z

Oh, that would imply that the USE_SSL define isn't defined, looking at url.c just lines above the case CURLOPT_SSL_OPTIONS: line. That would rather imply that you get a build completely without SSL?

webmaster128 · 2017-02-20T14:15:38Z

I guess USE_SSL is something internal, that CMakeLists.txt should set, right? What is the difference between USE_SSL and SSL_ENABLED?

I cannot really tell what is included in the build. Since my application code fails when doing the above configuration, even my HTTP tests without SSL fail. Is the any useful debugging that I can perform on the .a file to answer your question?

bagder · 2017-02-20T14:52:20Z

It is internal but it depends on one of the TLS enabled variables being set. See https://github.com/curl/curl/blob/master/lib/curl_setup.h#L618

webmaster128 · 2017-02-20T16:08:36Z

I guess there is a USE_DARWINSSL block missing in my auto-generated curl_config.h.

webmaster128 · 2017-02-20T16:12:01Z

CMakeLists.txt

+  if(CMAKE_USE_DARWINSSL AND NOT SSL_ENABLED)
+    find_library(COREFOUNDATION_FRAMEWORK "CoreFoundation")
+    find_library(SECURITY_FRAMEWORK "Security")
+    if(SECURITY_FRAMEWORK)


I think this block does not properly handle the absense of required libraries. What about

find_library(COREFOUNDATION_FRAMEWORK CoreFoundation) if (NOT COREFOUNDATION_FRAMEWORK) message(FATAL ERROR "CoreFoundation framework not found") endif() find_library(SECURITY_FRAMEWORK Foundation) if (NOT SECURITY_FRAMEWORK) message(FATAL ERROR "Security framework not found") endif() set(SSL_ENABLED ON) set(USE_DARWINSSL ON) list(APPEND CURL_LIBS "${COREFOUNDATION_FRAMEWORK}" "${SECURITY_FRAMEWORK}")

You're right (even though in practice they'll never be absent). I added the checks.

webmaster128

While testing this change, I found that there is DarwinSSL block missing in curl_config.cmake, see https://github.com/curl/curl/pull/1270/files

ligfx · 2017-02-21T00:36:39Z

Thanks so much for testing this out and investigating that issue, @webmaster128. I've cherry-picked your commit from the other PR into this one.

I also added another commit that checks that at most one SSL option is enabled. If multiple options are defined, it raises an error like this:

CMake Error at CMakeLists.txt:307 (message):
  Multiple SSL options specified: CMAKE_USE_DARWINSSL;CMAKE_USE_OPENSSL.
  Please pick at most one and disable the rest.

webmaster128 · 2017-02-21T07:13:31Z

CMakeLists.txt

  string(REGEX REPLACE "([a-zA-Z_][a-zA-Z0-9_]*)[\t ]*=[\t ]*([^\n]*)" "SET(\\1 \\2)" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})
-  string(REPLACE "�!�" "\n" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})
+  string(REPLACE "�!�" "\n" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})


I think you accidentally changed these lines. See #1271

I responded on your PR.

webmaster128 · 2017-02-21T07:14:32Z

CMakeLists.txt

+       message(FATAL ERROR "Security framework not found")
+    endif()
+
+    if(SECURITY_FRAMEWORK)


Now this check is obsolete

Good catch! This code looks much cleaner now, that was a great idea to disallow multiple SSL provider options.

webmaster128 · 2017-02-21T07:16:36Z

CMakeLists.txt

+
+    find_library(SECURITY_FRAMEWORK "Security")
+    if(NOT SECURITY_FRAMEWORK)
+       message(FATAL ERROR "Security framework not found")


There is an underscore missing (my fault, I guess). It is FATAL_ERROR, see https://cmake.org/cmake/help/v2.8.8/cmake.html#command%3amessage

Same for the block above.

It does the same thing regardless and aborts CMake, right? 👍 Fixed.

FATAL ERROR was a typo that I copied from my own project where I wrote it the wrong way

webmaster128 · 2017-02-21T07:19:20Z

Great! I found some minor other stuff. I'll test again soon

webmaster128 · 2017-02-21T16:17:36Z

CMakeLists.txt

+       message(FATAL_ERROR "Security framework not found")
+    endif()
+
+    if(SECURITY_FRAMEWORK)


The obsolete check is still in here

webmaster128 · 2017-02-21T16:19:02Z

CMakeLists.txt

+option(CMAKE_USE_MBEDTLS "Enable mbedTLS for SSL/TLS" OFF)
+if(CMAKE_USE_MBEDTLS)
+  find_package(MbedTLS)
+  if(MBEDTLS_FOUND)


This should fail hard and early when package MbedTLS was not found (like we discussed for CoreFoundation/Security)

Good catch, I see the pattern now. I'll also do the same thing for OpenSSL.

webmaster128 · 2017-02-21T16:21:00Z

CMakeLists.txt

  string(REGEX REPLACE "([a-zA-Z_][a-zA-Z0-9_]*)[\t ]*=[\t ]*([^\n]*)" "SET(\\1 \\2)" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})
-  string(REPLACE "�!�" "\n" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})
+  string(REPLACE "�!�" "\n" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})


When you replace to "" in 1075, line 1077 should be removed

I don't understand. I'm not changing line 1075/1077. These changes were accidental and I'll remove them from the PR.

Have a look at the full diff and you'll see that there is still a change that does not really make sense: https://github.com/curl/curl/pull/1228/files

... which is gone right now :)

webmaster128 · 2017-02-21T16:24:16Z

CMakeLists.txt

+  CMAKE_USE_OPENSSL
+  CMAKE_USE_MBEDTLS
+)
+if(enabled_ssl_options MATCHES ";")


Isn't there a way to get the length of a list for this check? Does this work in cmake 2.8? https://cmake.org/cmake/help/v3.0/command/list.html

There is, but that would add another command to call. I'll add a list length output parameter to collect_true.

webmaster128 · 2017-02-21T22:31:52Z

My runtime issue is gone now (just verified the current patch).

What now happens is the following:

// apply this PR onto current master and commit

$ git clean -xdf

$ cmake -DCMAKE_USE_DARWINSSL=1 -DHTTP_ONLY=1 -Wno-dev
// long healthy output ending with:
-- Enabled features: DarwinSSL IPv6 unix-sockets libz Largefile NTLM
-- Enabled protocols: HTTP HTTPS
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/me/myorg/curlwrapper/curl

$ cmake -DCMAKE_USE_DARWINSSL=1 -DHTTP_ONLY=1 -Wno-dev
CMake Warning at CMakeLists.txt:49 (message):
  the curl cmake build system is poorly maintained.  Be aware


-- curl version=[7.53.0-DEV]
Found *nroff option: -- -man
CMake Error at CMakeLists.txt:307 (message):
  Multiple SSL options specified: CMAKE_USE_DARWINSSL;CMAKE_USE_OPENSSL.
  Please pick at most one and disable the rest.


-- Configuring incomplete, errors occurred!

When you look at the cache, you see, that there are two CMAKE_USE_*SSL variables set:

//enable Apple OS native SSL/TLS
CMAKE_USE_DARWINSSL:BOOL=ON

//Use GSSAPI implementation (right now only Heimdal is supported
// with CMake build)
CMAKE_USE_GSSAPI:BOOL=OFF

//Use libSSH2
CMAKE_USE_LIBSSH2:BOOL=OFF

//Enable mbedTLS for SSL/TLS
CMAKE_USE_MBEDTLS:BOOL=OFF

//Use OpenSSL code. Experimental
CMAKE_USE_OPENSSL:BOOL=ON

This is because CMAKE_USE_OPENSSL is ON by default. It is not caugth in the first run because the option is set after the multiple ssl check is performed.

I think the most sustainable way to fix this is to default CMAKE_USE_OPENSSL to OFF.

ligfx · 2017-02-21T22:44:20Z

@webmaster128 Thanks for the further testing!

The Autoconf build defaults to enabling OpenSSL. I think enabling some form of SSL by default is a good idea since most users expect curl to be able to make secure connections.

I can move the option definitions to the top of "check for SSL" block, before the multiple SSL check is performed. In some ways that's cleaner than before, and makes it easier to do things like different defaults per-OS (if that's ever wanted). That would look like this:

option(CMAKE_USE_OPENSSL "Use OpenSSL code. Experimental" ON)
option(CMAKE_USE_MBEDTLS "Enable mbedTLS for SSL/TLS" OFF)
if(APPLE)
  option(CMAKE_USE_DARWINSSL "enable Apple OS native SSL/TLS" OFF)
endif()
if(WIN32)
  option(CMAKE_USE_WINSSL "enable Windows native SSL/TLS" OFF)
  cmake_dependent_option(CURL_WINDOWS_SSPI "Use windows libraries to allow NTLM authentication without openssl" ON
    CMAKE_USE_WINSSL OFF)
endif()

collect_true(enabled_ssl_options enabled_ssl_options_count
  CMAKE_USE_WINSSL
  CMAKE_USE_DARWINSSL
  CMAKE_USE_OPENSSL
  CMAKE_USE_MBEDTLS
)
if(enabled_ssl_options_count GREATER 1)
  message(FATAL_ERROR "Multiple SSL options specified: ${enabled_ssl_options}. Please pick at most one and disable the rest.")
endif()

if(CMAKE_USE_WINSSL)
  set(SSL_ENABLED ON)
  set(USE_SCHANNEL ON) # Windows native SSL/TLS support
  set(USE_WINDOWS_SSPI ON) # CMAKE_USE_WINSSL implies CURL_WINDOWS_SSPI
  list(APPEND CURL_LIBS "crypt32")
endif()
if(CURL_WINDOWS_SSPI)
  set(USE_WINDOWS_SSPI ON)
endif()

if(CMAKE_USE_DARWINSSL)
  find_library(COREFOUNDATION_FRAMEWORK "CoreFoundation")

Would that work for you?

webmaster128 · 2017-02-21T22:48:48Z

This means everyone using a different TLS provider has to explicitly disable OpenSSL, right? cmake -DCMAKE_USE_DARWINSSL=1 would not work anymore. I think the default should only be set when no other TLS provider is enabled.

Apart from that, the drafted version looks good to me.

ligfx · 2017-02-21T23:18:32Z

@webmaster128 Okay, I've updated the PR with that logic.

webmaster128 · 2017-02-22T10:36:00Z

All tests run green now :)

The only question I have left is, if there is an equivalent of --without-ca-bundle --without-ca-path from my old build script. Without this parameters I was having the issue that CA files where searched on a user's machine because they were present on the build machine. Is that in the scope of this PR?

bagder · 2017-02-22T10:38:32Z

I suggest to take that in a separate PR, as that's rather independent of specific TLS backend.

ligfx · 2017-02-22T20:16:41Z

I didn't realize I changed that behavior. I've changed the PR so it acts the same way as before (doesn't default to OpenSSL on Windows).

Lekensteyn

Looks mostly good, but add REQUIRED to find_package

Lekensteyn · 2017-02-24T23:34:43Z

CMakeLists.txt

-    set(CURL_WINDOWS_SSPI ON)
+  if(NOT OPENSSL_FOUND)
+    message(FATAL_ERROR "OpenSSL not found")
+  endif()


These three lines can be removed if you add "REQUIRED" to find_package

Good catch. Fixed both of these.

Lekensteyn · 2017-02-24T23:35:06Z

CMakeLists.txt

+  find_library(COREFOUNDATION_FRAMEWORK "CoreFoundation")
+  if(NOT COREFOUNDATION_FRAMEWORK)
+      message(FATAL_ERROR "CoreFoundation framework not found")
+  endif()


These three lines can be removed if you add REQUIRED to find_library (see cmake-commands manpage)

I don't think there is REQUIRED in find_library. At least it is not documented.

You're right, there is no REQUIRED for find_library

Lekensteyn · 2017-02-24T23:35:26Z

CMakeLists.txt

+if(CMAKE_USE_MBEDTLS)
+  find_package(MbedTLS)
+  if(NOT MBEDTLS_FOUND)
+    message(FATAL_ERROR "mbed TLS not found")
  endif()


REQUIRED to find_package

Lekensteyn · 2017-02-25T23:00:11Z

Hmm strange, why is CI failing here: https://travis-ci.org/curl/curl/jobs/205374812#L3906

Maybe it is an intermittent error. Can you try to update the INSTALL.cmake file with the updated state for SSL library support and re-trigger CI?

ligfx · 2017-02-26T00:06:15Z

@Lekensteyn Yeah it must be, none of the files touched in this PR are involved in that build process. I've updated the INSTALL.cmake file and re-pushed.

jay · 2017-02-26T00:11:52Z

Hmm strange, why is CI failing here: https://travis-ci.org/curl/curl/jobs/205374812#L3906

From what I understand pipelining there is a timing component in what order the data is received so occasionally they fail.

Lekensteyn

LGTM, still works on Arch Linux with OpenSSL.

Lekensteyn · 2017-02-26T11:12:10Z

@jay When merging these patches, should only the last commit message to include "Closes ..."? (Is something else needed?)

webmaster128 · 2017-02-26T21:10:15Z

Btw., is CMAKE_* really a naming convention that should be used for the public curl configuration API? As far as I can see, CMAKE_* variables usually come from the CMAKE system (CMAKE_SYSTEM_NAME, CMAKE_C_FLAGS and so on).

The name CMAKE_USE_OPENSSL was introduced in 2009 probably just to avoid a conflict with USE_OPENSSL.

Maybe those should better be prefixed with CURL_*

ligfx · 2017-02-27T03:38:56Z

@webmaster128 I totally agree. That's a good focus for another PR.

jay · 2017-02-27T08:20:33Z

When merging these patches, should only the last commit message to include "Closes ..."? (Is something else needed?)

Typically if it were me I would squash all commits or ask the reporter to do it. I see this is a few things in one PR and there are some distinct changes it may be useful leave separate, however this is more work.

'Closes' directive is never needed as a matter of protocol, it could be closed manually. What is important though is that we have some kind of a reference Closes/Fixes/Bug/Ref:, for example

Ref: https://github.com/curl/curl/pull/1228

Ideally something like that would near the end of each of these commit messages, so git blame nobody has to play detective. Off the top of my head I can't easily think of an automated way to append to all the commit messages at once.

Right now there is:

CMake: Add support for DarwinSSL
CMake: Reorganized SSL support, separated WinSSL and SSPI
CMake: Add mbedTLS support
CMake: at most one SSL library may be set
Add missing DarwinSSL block to curl_config.cmake
CMake: update INSTALL about new SSL lib support

At the very least the commit messages should be modified to meet the style format. That would involve changing the subjects to imperative present tense, adding missing area, ref, etc:

CMake: Add support for DarwinSSL
CMake: Reorganize SSL support, separate WinSSL and SSPI
CMake: Add mbedTLS support
CMake: Set at most one SSL library
CMake: Add missing DarwinSSL block to curl_config.cmake
CMake: update INSTALL about new SSL lib support

Then after rebase make sure to confirm that you're the committer on all of them, and the authors are left intact.

If it were me I'd lose at least these squashable 2:

Pick Add missing DarwinSSL block to curl_config.cmake into CMake: Add support for DarwinSSL and add a Assisted-by: Simon Warta line to that commit message.

Squash CMake: update INSTALL about new SSL lib support into CMake: at most one SSL library may be set.

CMake: Add support for DarwinSSL
CMake: Reorganize SSL support, separate WinSSL and SSPI
CMake: Add mbedTLS support
CMake: Set at most one SSL library

webmaster128 · 2017-02-27T08:39:44Z

I totally agree. That's a good focus for another PR.

I can understand that you want this (excellent) piece of work to be done, but my concern is that this introduces two new public variables in a naming format that is at least confusing. As soon as this is in, we need to think about compatibility and I don't think it is worth carrying around compatibility code for a decade just because CMAKE_USE_DARWINSSL and CMAKE_USE_MBEDTLS have been used for one or two weeks, maybe including a release.

jzakrzewski · 2017-02-27T08:53:02Z

I think we should do this once and good. The names for the options used in command line should be short and simple - without any prefix. After all the user is building curl and he/she and we know that so why should we force them to type those prefixes?

So I'd say - make a separate PR, cleanup all names, break the compatibility once but do it the right way.

Lekensteyn · 2017-02-28T17:14:10Z

@ligfx Can you squash the commits according to the suggestions from @jay?

@webmaster128 @jzakrzewski There are more inconsistent names (CMAKE_USE_GSSAPI, CMAKE_USE_LIBSSH2), feel free to propose a patch tot rename them (and for backwards compat, the default for the renamed options could use the former).

Maybe it would be even better to introduce a separate CMakeOptions.txt to keep all options in one place. See for example https://github.com/nghttp2/nghttp2/blob/master/CMakeOptions.txt

This is closer to how configure.ac does it Ref: curl#1228

Assisted-by: Simon Warta <simon@kullo.net> Ref: curl#1228

Ref: curl#1228

ligfx · 2017-02-28T22:27:03Z

@Lekensteyn done

This is closer to how configure.ac does it Ref: #1228

Assisted-by: Simon Warta <simon@kullo.net> Ref: #1228

Ref: #1228

Lekensteyn · 2017-03-05T13:49:17Z

Merged via:
f85ff14 CMake: Reorganize SSL support, separate WinSSL and SSPI
eb19e89 CMake: Add DarwinSSL support
c6a9746 CMake: Add mbedTLS support
813263d CMake: Set at most one SSL library

Thanks for your contribution @ligfx!

jay added the cmake label Jan 26, 2017

Lekensteyn reviewed Feb 1, 2017

View reviewed changes

ligfx force-pushed the cleanup branch from 007a5dd to 86bfc23 Compare February 3, 2017 17:01

webmaster128 reviewed Feb 19, 2017

View reviewed changes

webmaster128 reviewed Feb 20, 2017

View reviewed changes

webmaster128 mentioned this pull request Feb 20, 2017

Add missing DarwinSSL block to curl_config.cmake #1270

Closed

webmaster128 suggested changes Feb 20, 2017

View reviewed changes

ligfx force-pushed the cleanup branch from 86bfc23 to 2ee5241 Compare February 21, 2017 00:38

webmaster128 suggested changes Feb 21, 2017

View reviewed changes

ligfx force-pushed the cleanup branch from 2ee5241 to 01484fc Compare February 21, 2017 09:37

webmaster128 suggested changes Feb 21, 2017

View reviewed changes

ligfx force-pushed the cleanup branch from 01484fc to b9f5f62 Compare February 21, 2017 22:33

ligfx force-pushed the cleanup branch from b9f5f62 to 85d1a8f Compare February 21, 2017 23:18

ligfx force-pushed the cleanup branch from 85d1a8f to c42345e Compare February 21, 2017 23:22

webmaster128 approved these changes Feb 22, 2017

View reviewed changes

ligfx force-pushed the cleanup branch from c42345e to 0515cff Compare February 22, 2017 20:16

Lekensteyn suggested changes Feb 24, 2017

View reviewed changes

ligfx force-pushed the cleanup branch from 0515cff to d94ce85 Compare February 25, 2017 21:57

ligfx force-pushed the cleanup branch from d94ce85 to 97dcbbe Compare February 26, 2017 00:06

Lekensteyn approved these changes Feb 26, 2017

View reviewed changes

ligfx added 4 commits February 28, 2017 14:11

CMake: Reorganize SSL support, separate WinSSL and SSPI

2258006

This is closer to how configure.ac does it Ref: curl#1228

CMake: Add DarwinSSL support

fb08797

Assisted-by: Simon Warta <simon@kullo.net> Ref: curl#1228

CMake: Add mbedTLS support

2823c63

Ref: curl#1228

CMake: Set at most one SSL library

bcb6c4a

Ref: curl#1228

ligfx force-pushed the cleanup branch from 97dcbbe to bcb6c4a Compare February 28, 2017 22:18

Lekensteyn pushed a commit that referenced this pull request Mar 5, 2017

CMake: Reorganize SSL support, separate WinSSL and SSPI

f85ff14

This is closer to how configure.ac does it Ref: #1228

Lekensteyn pushed a commit that referenced this pull request Mar 5, 2017

CMake: Add DarwinSSL support

eb19e89

Assisted-by: Simon Warta <simon@kullo.net> Ref: #1228

Lekensteyn pushed a commit that referenced this pull request Mar 5, 2017

CMake: Add mbedTLS support

c6a9746

Ref: #1228

Lekensteyn pushed a commit that referenced this pull request Mar 5, 2017

CMake: Set at most one SSL library

813263d

Ref: #1228

Lekensteyn closed this Mar 5, 2017

lock bot locked as resolved and limited conversation to collaborators May 24, 2018

CMake: Support for DarwinSSL and mbedTLS + cleanup #1228

CMake: Support for DarwinSSL and mbedTLS + cleanup #1228

Conversation

ligfx commented Jan 26, 2017

mention-bot commented Jan 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ligfx Feb 3, 2017 • edited Loading

Choose a reason for hiding this comment

webmaster128 left a comment

Choose a reason for hiding this comment

webmaster128 Feb 19, 2017 • edited Loading

Choose a reason for hiding this comment

webmaster128 commented Feb 20, 2017 • edited Loading

bagder commented Feb 20, 2017

webmaster128 commented Feb 20, 2017

bagder commented Feb 20, 2017

webmaster128 commented Feb 20, 2017 • edited Loading

bagder commented Feb 20, 2017

webmaster128 commented Feb 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmaster128 left a comment

Choose a reason for hiding this comment

ligfx commented Feb 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmaster128 commented Feb 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmaster128 commented Feb 21, 2017 • edited Loading

ligfx commented Feb 21, 2017

webmaster128 commented Feb 21, 2017 • edited Loading

ligfx commented Feb 21, 2017

webmaster128 commented Feb 22, 2017

bagder commented Feb 22, 2017

ligfx commented Feb 22, 2017

Lekensteyn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lekensteyn commented Feb 25, 2017 • edited Loading

ligfx commented Feb 26, 2017

jay commented Feb 26, 2017

Lekensteyn left a comment

Choose a reason for hiding this comment

Lekensteyn commented Feb 26, 2017

webmaster128 commented Feb 26, 2017

ligfx commented Feb 27, 2017

jay commented Feb 27, 2017 • edited Loading

webmaster128 commented Feb 27, 2017

jzakrzewski commented Feb 27, 2017

Lekensteyn commented Feb 28, 2017

ligfx commented Feb 28, 2017

Lekensteyn commented Mar 5, 2017

ligfx Feb 3, 2017 •

edited

Loading

webmaster128 Feb 19, 2017 •

edited

Loading

webmaster128 commented Feb 20, 2017 •

edited

Loading

webmaster128 commented Feb 20, 2017 •

edited

Loading

webmaster128 commented Feb 21, 2017 •

edited

Loading

webmaster128 commented Feb 21, 2017 •

edited

Loading

Lekensteyn commented Feb 25, 2017 •

edited

Loading

jay commented Feb 27, 2017 •

edited

Loading