New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

php7 + opcache + mongodb: Query and php array is getting changed randomly. #216

Closed
cb-msales opened this Issue Jul 20, 2016 · 44 comments

Comments

Projects
None yet
8 participants
@cb-msales

cb-msales commented Jul 20, 2016

We found some strange behaviour where queries sent to mongodb don't match what we intended to send.

After digging deeper we found that under load with multiple requests handeled concurrently by php-fpm, the system randomly starts changing the php array (!).
This only happens when the array is sent to the mongodb find operation (other operations not tested).

We have only seen it changing the array when it has been sent to mongodb. Not calling ->find() does not produce the error.

Also opcache need to be enabled for the file where the query array is defined. With opcache disabled it will not fail.

The array is changed in a way that 'static' contents seem to be removed. We don't know if any more is getting replaced or changed. For example the part '[10, 12]' is replaced with NULL in our test case.
Replacing one value with dynamic content eliminates the error too: repalce for example 12 with (int)$_GET['id'], call the test script with parameter test.php?id=10 and it will NOT fail.

When using json_decode(json_encode($test), true) before submitting the data to ->find it will NOT fail and not change any contents of $test.

The database and selection does not exist in our test case.

We have tested with
ext-mongodb 1.1.8
mongodb/mongodb 1.0.2
PHP 7.0.8-4

Below the test script, run with apache benchmark


require __DIR__ . '/../vendor/autoload.php';

$test = ['test' => ['$in' => [10, 12]]];

$client = new MongoDB\Client();
$client->selectCollection('a','b')->find($test);

file_put_contents('/tmp/mongodb-errors.log', (int)(serialize($test) == 'a:1:{s:4:"test";a:1:{s:3:"$in";a:2:{i:0;i:10;i:1;i:12;}}}'), FILE_APPEND);

Run ab test:
(optional) > sudo service php-fpm restart;

ab -c 40 -n 1000 http://your.domain.goes.here/test.php

Example result: 1 = success, array not changed, 0 = error, array has been changed. '[10, 12]' has been nulled.

11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
11111111111111111111111111111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000
00000000000000000000000100000000000000000000000000000000000000000000000000111111111111111111111111111111111111111111111111
11100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000111111111111111000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000001111111```

@cb-msales cb-msales changed the title from php7 + opcache + mongodb: Query and php array is getting changed randomply. to php7 + opcache + mongodb: Query and php array is getting changed randomly. Jul 20, 2016

@jmikola

This comment has been minimized.

Member

jmikola commented Jul 20, 2016

This seems like it may be a duplicate of mongodb/mongo-php-driver#293, which was caused by APCu. Can you read through that issue and confirm?

@cb-msales

This comment has been minimized.

cb-msales commented Jul 20, 2016

php -i | grep -i apcu
does not give me any result. I think we are not using APCu.

@cb-msales

This comment has been minimized.

cb-msales commented Jul 21, 2016

If you need any help reproducing the issue let me know. I'll help where I can as this is an important issue for us.

@jmikola

This comment has been minimized.

Member

jmikola commented Jul 21, 2016

php -i | grep -i apcu does not give me any result. I think we are not using APCu.

I suppose it might be helpful to get a record of all PHP extensions being used. Or perhaps all php -i output in a gist in case some INI settings are also relevant.

Replacing one value with dynamic content eliminates the error too: replace for example 12 with (int)$_GET['id'], call the test script with parameter test.php?id=10 and it will NOT fail.

When using json_decode(json_encode($test), true) before submitting the data to ->find it will NOT fail and not change any contents of $test.

This sounds quite bizarre, as I don't see how modifying the value beforehand would affect the driver's internal behavior. I imagine it may have implications for opcache, though.

When the MongoDB\Driver\Query is constructed, the argument is immediately serialized to BSON. That BSON is what is ultimately sent to the server. Can you use server-side profiling to log queries sent the to server and compare whether they are also be corrupted? The comment option to find() may be used to make it easier to track these queries.

Also, I noticed that you were dumping serialize($test) after calling find(). I might suggest moving those comparisons to the lines immediately before and after the library constructs a MongoDB\Driver\Query object (here) for testing purposes.

@cb-msales

This comment has been minimized.

cb-msales commented Jul 22, 2016

More tests made:

Before and after the find call:

require __DIR__ . '/../vendor/autoload.php';

$test = ['test' => ['$in' => [10, 12]]];

$client = new MongoDB\Client();
$a = (int)(\serialize($test) == 'a:1:{s:4:"test";a:1:{s:3:"$in";a:2:{i:0;i:10;i:1;i:12;}}}');
$client->selectCollection('a','b')->find($test);
$b = (int)(\serialize($test) == 'a:1:{s:4:"test";a:1:{s:3:"$in";a:2:{i:0;i:10;i:1;i:12;}}}');
file_put_contents('/tmp/mongodb-errors.log', $a . '-' . $b . " ", FILE_APPEND);

Results:

looks like error manifested like this

1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-0 1-0 1-0 1-0 1-0 1-0 1-0 1-0 1-0 1-0 1-0 1-0 1-0 1-0 1-0 1-0 0-0 1-0 1-0 0-0 0-0 1-0 1-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 1-0 0-0 0-0 0-0 0-0 0-0 0-0 1-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0

and in a laterrun the error disappeared again:

0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-1 0-1 0-1 0-1 1-1 0-1 0-1 1-1 0-1 0-1 1-1 1-1 0-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 0-1 0-1 0-1 0-1 0-1 1-1 1-1 1-1 1-1 1-1 1-1 0-1 0-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1


And now as you suggested before and after MongoDB\Driver\Query is constructed

$a = (int)(\serialize($this->filter) == 'a:1:{s:4:"test";a:1:{s:3:"$in";a:2:{i:0;i:10;i:1;i:12;}}}');
$ret = new Query($this->filter, $options);
$b = (int)(\serialize($this->filter) == 'a:1:{s:4:"test";a:1:{s:3:"$in";a:2:{i:0;i:10;i:1;i:12;}}}');
file_put_contents('/tmp/mongodb-errors.log', $a . '-' . $b . " ", FILE_APPEND);
return $ret;

first run:

1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-0 0-0 0-0 0-0 1-1 0-0 0-0 0-0 0-0 0-0 0-0 0-0 0-0

second run:

0-0 0-0 0-0 0-0 0-1 1-0 0-0 0-0 0-0 0-0

another run after php-fpm restart

0-0 0-0 0-0 0-0 0-0 0-1 0-1 1-1 1-1 1-1 [ more 1-1 here] 1-1 1-1 1-0 1-0 0-0 0-0 0-0 0-0


logging queries in mongodb server:

'comment' => 'abc' used:

After running I dumped the result into abc.log

less /var/log/mongodb/mongod.log | grep abc > abc.log

The log file is attached to this issue.

In line 199 is the first error happening:

"{ $comment: "abc", $query: { test: { $in: [] } } }"

The values inside the array have been removed.

It doesn't look like the corrupted php $test array is NOT exactly linked to the number of wrong queries. But still somewhat related.

@cb-msales

This comment has been minimized.

cb-msales commented Jul 22, 2016

Here my list of extensions:

Core
PDO
Phar
Reflection
SPL
SimpleXML
Zend OPcache
calendar
cgi-fcgi
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gd
gettext
hash
iconv
json
libxml
mbstring
mongodb
openssl
pcre
pdo_pgsql
pgsql
posix
readline
session
sockets
standard
tokenizer
wddx
xml
xmlreader
xmlwriter
xsl
zlib

@cb-msales

This comment has been minimized.

cb-msales commented Jul 26, 2016

Is anyone able to reproduce the issue?

@jmikola

This comment has been minimized.

Member

jmikola commented Aug 4, 2016

In both of your tests (first around Collection::find() and secondly around the Query construction) where you were comparing the serialized filter array to an expected string, you appear to have a mix of both comparisons being false at different times. That hints at some sort of corruption before we even get to the driver code.

If Query::__construct() was at fault, I might expect your output to be a mix of 1-1 and 1-0, but I would not expect the first comparison to ever be false. Essentially, this is failing for you:

$test = ['test' => ['$in' => [10, 12]]];
$a = (int)(\serialize(['test' => ['$in' => [10, 12]]]) == 'a:1:{s:4:"test";a:1:{s:3:"$in";a:2:{i:0;i:10;i:1;i:12;}}}');

Have you tried testing that on its own without using the driver at all?

The log file is attached to this issue.

I don't see any files attached to your earlier posts. You may want to double-check that.

@cb-msales

This comment has been minimized.

cb-msales commented Aug 5, 2016

I agree, the problem seems to persist across multiple calls. It sometimes comes and sometimes goes away again. And again it only happens when multiple calls are procesed concurrently..

Without the mongodb driver or even without calling ->find($test); method it does not happen.

@cb-msales

This comment has been minimized.

cb-msales commented Aug 5, 2016

don't see any files attached to your earlier posts. You may want to double-check that.

Here we go:

abc.log.txt

@Visualq

This comment has been minimized.

Visualq commented Aug 13, 2016

I have a similar problem which I think is related to this. Also noticed that when I disable opcache it works without a problem.

The query which fails is: $table->find(['_type' => ['$in' => ['website', 'rootnode', 'iqcms']]]); It started to return 0 results. I rewrote the query to test if it mattered to: ['$or' => [['_type' => 'website'], ['_type' => 'iqcms'], ['_type' => 'iqcms']]];

The latter query eventually started producing the following exception:

Fatal error: Uncaught MongoDB\Driver\Exception\ConnectionException: $and/$or/$nor must be a nonempty array in /var/www/iqcms-1.6/vendor/mongodb/mongodb/src/Operation/Find.php:180

I test it by doing a ab -n 4000 -c 20 http://dgd70.localhost.iqcms/ and after 500-5000 requests (seems random) it starts to produce exceptions because the query isn’t returning any results. The code base is a bit large and I haven’t isolated the query to see if I can reproduce it with a smaller set of code.

My opcache config:
[opcache]
zend_extension="opcache.so"
opcache.enable=1
opcache.enable_cli=1
opcache.memory_consumption=256
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=7963
opcache.max_wasted_percentage=5
opcache.use_cwd=1
opcache.validate_timestamps=1
opcache.revalidate_freq=60
opcache.fast_shutdown=1

I encounter the problem with the 1.1.8 driver aswell as the 1.2.0alpha.

Let me know if I need to provide more data.

@cb-msales

This comment has been minimized.

cb-msales commented Aug 15, 2016

Seems like this is caused at a lower level than the php library, possibly some memory corruption going on here? I created that issue in the mongodb-php-driver repository to get some help from there.

@jmikola

This comment has been minimized.

Member

jmikola commented Aug 15, 2016

@cb-msales: Does your opcache configuration agree with @Visualq?

Would either of you be able to attempt reproducing the problem with the Zend memory manager disabled? This will require setting the USE_ZEND_ALLOC environment variable to 0. To ensure that the environment variable is respected, you should see "Zend Memory Manager" marked as "disabled" in phpinfo().

This blog post may be helpful for specifying an environment variable in PHP-FPM.

@derickr suggested this on the off chance that Zend MM is hiding the underlying issue. Reproducing without it may provide a better chance at exposing the root cause.

@Visualq

This comment has been minimized.

Visualq commented Aug 15, 2016

@jmikola It seems that disabling the Zend Memory Manager does not solve the problem. The problem keeps occuring.

@derickr

This comment has been minimized.

Member

derickr commented Aug 15, 2016

Sorry, @Visualq, we had not expected the problem to go away with this, but rather, it should be easier to deterministically reproduce the problem.

@Visualq

This comment has been minimized.

Visualq commented Aug 16, 2016

Ah yes, I didn't properly read the question. To properly answer the question: I was able to reproduce the problem with Zend Memory Manager disabled. Is there anything I can provide to help determining the cause? The data used to make this problem appear is sensitive and therefore I can't make it publicly available.

@cb-msales

This comment has been minimized.

cb-msales commented Aug 25, 2016

@Visualq Could you try the code snippet I posted at the beginning? Let's see if it is reproducible.

@cb-msales

This comment has been minimized.

cb-msales commented Aug 25, 2016

@jmikola Sorry for my late reply. I used the standard opcache config (no parameters set in php.ini) and in a second test I also used Visualq's exact config. The bug appeared every time.

@Visualq

This comment has been minimized.

Visualq commented Aug 25, 2016

@cb-msales I managed to reproduce the problem with the snippet you provided. took around 1500 requests before the 0's started to appear. However after the first 0 appears the array will not be back. It will fail every time.

@Visualq

This comment has been minimized.

Visualq commented Sep 30, 2016

Any progress on this problem? It prevents me from using PHP 7 + Mongo + opcache in production.

@cb-msales

This comment has been minimized.

cb-msales commented Sep 30, 2016

Hi, @Visualq have you tried to json_encode and directly decode your variables passed to mongodb calls?

It seems to work fine for us as a temporary workaround because the variables contents don't get optimized away anymore. No reference can be passed to MongoDD this way.

->find( json_decode(json_encode($query), true) );

Of course that's just a temporary solution and doesn't replace a fix of the underlying issue.

@Visualq

This comment has been minimized.

Visualq commented Sep 30, 2016

Hi @cb-msales, Thank you, but as much as I appreciate the workaround I have feeling this problem will occur at more places which will lead to unexplainable events causing me to spend to much debug time ;). I think I rather wait for the proper solution. I will try the json_decode/encode trick later on just to see if that works.

@loonytoons

This comment has been minimized.

loonytoons commented Oct 4, 2016

We also ran into this issue, with queries using the aggregation framework.
Under load we'd find no results being returned by the query, then after around 10 mins or so results would appear again before disappearing a short while later.
Turning off the opcache has fixed this issue but really isn't an ideal solution as its impacting performance.

We're running:

  • php 7.0.11-1+deb.sury.org~xenial+1
  • mongodb 1.1.8
  • libmongoc 1.3.5
  • 3.2.10

Happy to send over any more info that might help track down this issue :-)

@quibusus

This comment has been minimized.

quibusus commented Nov 8, 2016

I can confirm similar issue with aggregate.

When running aggregate queries with PHP7, we start getting following exceptions
exception: Expression $cond takes exactly 3 arguments. 0 were passed in.

  • seems like query is corrupted before hitting the database
  • same queries have random results (either result or exception)
  • problem appears under high load only
  • Apache restart helps for 10-20 mins, then the exceptions reappear
  • PHP7 + opcache
@mbretter

This comment has been minimized.

mbretter commented Nov 8, 2016

Today, we ran into exactly the same problem, we have 2 servers behind a load balancer, one server started returning no results for a query, which works on the second server. Both severs are using the same database.

After a restart of php-fpm the problem disappeared.

php 7.0.8-0ubuntu0.16.04.3 + opcache
mongodb 1.1.6 stable (installed from pecl)
mongodb server 2.6.11

@moderndeveloperllc

This comment has been minimized.

moderndeveloperllc commented Nov 29, 2016

@derickr, @jmikola FWIW, This is still occurring on the just released mongodb 1.2.0 extension w/ mongod 3.2.10 using the original code from @cb-msales July 20 comment. Commenting out the find() negates the issue. Doing the json_decode(json_encode($query)) workaround also negates the issue.

$ php -v
PHP 7.1.0RC6 (cli) (built: Nov  9 2016 15:11:15) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.1.0-dev, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.1.0RC6, Copyright (c) 1999-2016, by Zend Technologies
    with Xdebug v2.5.0-dev, Copyright (c) 2002-2016, by Derick Rethans

$ php --ri mongodb
mongodb
mongodb support => enabled
mongodb version => 1.2.0
mongodb stability => stable
libmongoc version => 1.5.0
libbson version => 1.5.0

Directive => Local Value => Master Value
mongodb.debug => no value => no value
@moderndeveloperllc

This comment has been minimized.

moderndeveloperllc commented Nov 29, 2016

@cb-msales I was also able to get the correct results if I reset the opcache with opcache_reset(); at the head of the script. Not optimal at all since it negates one of the big benefits of PHP 7.

@jmikola

This comment has been minimized.

Member

jmikola commented Nov 30, 2016

@moderndeveloperllc: Thanks for confirming. I've opened PHPC-848 to track this and will set aside some time next week to investigate. I'd also like to attempt reproduction with PHP 7.1, which should be released by then.

@moderndeveloperllc

This comment has been minimized.

moderndeveloperllc commented Nov 30, 2016

@jmikola Good luck with that! I spent some time yesterday playing around on the userland end of things.

  1. I did an opcache_invalidate($script, true) for each of the 13 scripts called (including composer scripts) and the issue still occurs, so the issue is almost certainly driver code.
  2. I added opcache_get_status() to the logging output. Nothing popped out as being different for working/nonworking calls - only the hit counter.
  3. Issue only happened with concurrent threads which means that the file_put_content() logs are not in execution order so hopefully you have better tools.
  4. Nice to know 7.1 will get here before the new year!
@jmikola

This comment has been minimized.

Member

jmikola commented Nov 30, 2016

Issue only happened with concurrent threads which means that the file_put_content() logs are not in execution order so hopefully you have better tools.

@moderndeveloperllc: what's the relation to "concurrent threads". If I understand the thread correctly, folks have reproduced this with PHP FPM, where each worker handles a single request at a time. On that note, can you share your PHP and FPM configurations (i.e. php-fpm.conf and the www.conf pool configuration)?

@Visualq

This comment has been minimized.

Visualq commented Nov 30, 2016

@jmikola Thank you for looking into this bug.

I took the liberty testing the bug with 1.2 driver and PHP 7.1 RC6. I managed to reproduce the problem quite quickly.

I attached the php.ini and php-fpm.conf which I'm using to produce the bug. (www.conf is included in the fpm config.)

php.ini.txt
php-fpm.conf.txt

@jmikola

This comment has been minimized.

Member

jmikola commented Nov 30, 2016

@Visualq: Thanks, that should help quite a bit.

@moderndeveloperllc

This comment has been minimized.

moderndeveloperllc commented Nov 30, 2016

@jmikola I was just testing with the original setup that used ab. If I set the concurrent threads down to 1 (ab -c 1 -n 1000), I didn't see the issue in the first thousand. If the issue happening with PHP-FPM, then perhaps 'concurrency' is a false criterion.

@Visualq

This comment has been minimized.

Visualq commented Nov 30, 2016

@moderndeveloperllc I couldn't reproduce the issue either using 1 thread for 100k requests.

@jmikola

This comment has been minimized.

Member

jmikola commented Nov 30, 2016

@moderndeveloperllc, @Visualq: Apologies for assuming PHP FPM was being used here. To confirm, the reproductions are with Apache and mod_php?

@Visualq

This comment has been minimized.

Visualq commented Nov 30, 2016

@jmikola I'm using nginx 1.11.6 with php fpm. Not Apache.

@jmikola

This comment has been minimized.

Member

jmikola commented Nov 30, 2016

@Visualq: When you said "I couldn't reproduce the issue either using 1 thread for 100k requests", were you referring to a single FPM worker? I may be a bit confused because it's my understanding that each FPM workers are separate processes, not threads, and each only serves one request at a time. The only concurrency is if a pool contains multiple workers. The relevant bits from your config are:

pm = dynamic
pm.max_children = 64
pm.start_servers = 6
pm.min_spare_servers = 4
pm.max_spare_servers = 8
@Visualq

This comment has been minimized.

Visualq commented Nov 30, 2016

@jmikola ah no I meant I was using ab -c 1 -n 100000. I didn't change the fpm config.

@jmikola

This comment has been minimized.

Member

jmikola commented Dec 6, 2016

Just to follow up, I've been able to reproduce the issue locally with the provided configuration files and scripts and am actively investigating the cause. In addition to the library example, I've reproduced it with the driver directly (as expected) and also with MongoDB\BSON\fromPHP(), which hints that BSON encoding is to blame.

Library Query

<?php

require __DIR__ . '/vendor/autoload.php';

$test = ['test' => ['$in' => [10, 12]]];

$client = new MongoDB\Client;
$client->selectCollection('a','b')->find($test);

$result = (int)(serialize($test) == 'a:1:{s:4:"test";a:1:{s:3:"$in";a:2:{i:0;i:10;i:1;i:12;}}}');
file_put_contents('/tmp/mongodb-errors.log', $result, FILE_APPEND);

Extension Query

<?php

$test = ['test' => ['$in' => [10, 12]]];

$query = new MongoDB\Driver\Query($test);
$manager = new MongoDB\Driver\Manager;
$cursor = $manager->executeQuery('a.b', $query);

$result = (int)(serialize($test) == 'a:1:{s:4:"test";a:1:{s:3:"$in";a:2:{i:0;i:10;i:1;i:12;}}}');
file_put_contents('/tmp/mongodb-errors.log', $result, FILE_APPEND);

Extension BSON Encoding

<?php

$test = ['test' => ['$in' => [10, 12]]];
$bson = MongoDB\BSON\fromPHP($test);

$result = (int)(serialize($test) == 'a:1:{s:4:"test";a:1:{s:3:"$in";a:2:{i:0;i:10;i:1;i:12;}}}');
file_put_contents('/tmp/mongodb-errors.log', $result, FILE_APPEND);
@jmikola

This comment has been minimized.

Member

jmikola commented Dec 7, 2016

@cb-msales, @Visualq, @moderndeveloperllc: I'd appreciate it if you guys could test out the patch in mongodb/mongo-php-driver#481 and report back on whether that addresses the problem. It appears to fix the issue on my local environment (testing several rounds of 200k requests through ab), which is a 3-worker FPM deployment with @Visualq's OPcache configuration in effect.

I've open the PR against the v1.1 branch, but it will cleanly apply to master as well if you'd like to test on 1.2. If it does resolve the issue, I will get this released ASAP as 1.1.10 and 1.2.1.

@Visualq

This comment has been minimized.

Visualq commented Dec 7, 2016

Hi @jmikola, I applied the patch to the master branch and ran a few tests. It appears to be working! no more abnormalities.

In the patch you add an equivalent macro for working with the apply counter for PHP 5.x however I could not reproduce this problem with PHP 5.6 (pre-patch).

@jmikola

This comment has been minimized.

Member

jmikola commented Dec 7, 2016

In the patch you add an equivalent macro for working with the apply counter for PHP 5.x however I could not reproduce this problem with PHP 5.6 (pre-patch).

The macro for PHP 5.x always evaluates to true, since it has no notion of immutable arrays. It was just easier to keep a macro in place instead of requiring #if blocks around the code in bson.c for PHP 7.x and PHP 5.x.

@Visualq

This comment has been minimized.

Visualq commented Dec 7, 2016

Got it, thanks!

@jmikola

This comment has been minimized.

Member

jmikola commented Dec 7, 2016

1.1.10 and 1.2.1 have been released with the patch. In addition to addressing the OPcache issue, we resolved a bug with detecting circular references in arrays and objects, which is also relevant to PHP 5.x users.

@jmikola jmikola closed this Dec 7, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment