Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HHVM fixes - tests pass hunting [WIP] #165

Merged
merged 5 commits into from Sep 22, 2014
Merged

Conversation

milo
Copy link
Member

@milo milo commented Sep 4, 2014

One last assertion is not passing due to #162. Thinking about fix/workaround.

1st commit note: HHVM 3.3 should be out next Thursday.

$this->findTests(__DIR__ . '/find-tests/*.phptx');
Assert::count(1, $this->jobs);
});
if (defined('HHVM_VERSION')) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hack should not be needed anymore in future, since facebook/hhvm#1203 is fixed (currently only in master, but hopefully also in 3.3.0).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Majkl578 Thanks for the tip. I'm still waiting for HHVM 3.3.0 release to test all properly and release Tester 1.3.0.

@milo
Copy link
Member Author

milo commented Sep 21, 2014

Few days since HHVM 3.3.0 is out. I'll refator this PR soon //cc @Majkl578

@Majkl578
Copy link
Contributor

I upgraded HHVM on my VPS (which immediately got broken because of hhvm/packaging#72).
This is what I got after running tester under HHVM 3.3.0 (without your patch).

$ ./Tester/tester -p hhvm tests/
 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v1.2.0

Note: No php.ini is used.
PHP 5.6.99-hhvm | 'hhvm' --php | 33 threads

FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF..........

-- FAILED: tester/tests/Assert.contains.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.count.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.equal.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.equal.recursive.phpt [method=testSimple]
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.error.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.match.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.match.regexp.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.equal.recursive.phpt [method=testMultiple]
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.equal.recursive.phpt [method=testDeep]
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.equal.recursive.phpt [method=testCross]
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.same.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.true.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.type.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/DataProvider.load.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/CommandLine.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/DataProvider.parseAnnotation.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/DomQuery.fromHtml.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Collector.start.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.equal.recursive.phpt [method=testThirdParty]
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.exception.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.false.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.nan.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.null.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Runner.find-tests.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Runner.edge.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Runner.multiple-fails.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/ZendPhpExecutable.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Helpers.parseDocComment.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/TestCase.dataProvider.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/TestCase.invalidMethods.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/HhvmPhpInterpreter.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/TestCase.annotationThrows.syntax.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/TestCase.basic.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/TestCase.order.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Runner.annotations.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/TestCase.annotationThrows.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/TestCase.invalidProvider.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Runner.multiple.phpt
   Exited with error code 134 (expected 0)


FAILURES! (48 tests, 38 failures, 11.8 seconds)

$ hhvm --version
HipHop VM 3.3.0 (rel)
Compiler: tags/HHVM-3.3.0-0-g0a3cfb87b8a353fc7e1d15374f4adc413e37aba9
Repo schema: 9a391d9a03e15fccba1cde6d35c05b7cdd380238
Extension API: 20140829

Figured out it is related to parallel tests run (and it was working on 3.2!), so this is what I got with single thread (hell, HHVM is so painfully slow in CLI):

$ ./Tester/tester -p hhvm -j 1 tests/
 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v1.2.0

Note: No php.ini is used.
PHP 5.6.99-hhvm | 'hhvm' --php | 1 thread

...................s.................FFF.......s

-- FAILED: tester/tests/Runner.find-tests.phpt
   Exited with error code 255 (expected 0)

   Fatal error: Call to undefined method Closure$;820474927$de7f725ec58d68642d3a623a5892688$::bindTo() in /tmp/tester/Tester/Framework/Assert.php on line 436

-- FAILED: tester/tests/Runner.multiple-fails.phpt
   Exited with error code 255 (expected 0)

   Fatal error: Uncaught exception 'Tester\AssertException' with message ''Cannot list TestCase methods in file '/tmp/tester/tests/multiple-fails...' should match 'Error: This test forgets to execute an assertion.'' in /tmp/tester/Tester/Framework/Assert.php:425
   Stack trace:
   #0 /tmp/tester/Tester/Framework/Assert.php(397): Tester\Assert::fail()
   #1 /tmp/tester/tests/Runner.multiple-fails.phpt(51): Tester\Assert::match()
   #2 {main}
   Failed: 'Cannot list TestCase methods in file '/tmp/tester/tests/multiple-fails...' should match
       ... 'Error: This test forgets to execute an assertion.'

   diff '/tmp/tester/tests/output/Runner.multiple-fails.expected' '/tmp/tester/tests/output/Runner.multiple-fails.actual'

   in tester/tests/Runner.multiple-fails.phpt(51) Tester\Assert::match()

-- FAILED: tester/tests/Runner.multiple.phpt
   Exited with error code 255 (expected 0)

   Fatal error: Call to undefined method Closure$;89775627$f1f311ad7b10fa4ba36b2ab43219729a$::bindTo() in /tmp/tester/Tester/Framework/Assert.php on line 436


FAILURES! (48 tests, 3 failures, 2 skipped, 47.7 seconds)

And here for this PR, first parallel run, then serial run:

$ ./Tester/tester -p hhvm tests/
 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v1.2.0

Note: No php.ini is used.
PHP 5.6.99-hhvm | 'hhvm' --php -d hhvm.log.always_log_unhandled_exceptions=false  | 33 threads

FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFs..FFFFFF......

-- FAILED: tester/tests/Assert.equal.phpt
   Exited with error code 1 (expected 0)

-- FAILED: tester/tests/Assert.equal.recursive.phpt [method=testSimple]
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.equal.recursive.phpt [method=testMultiple]
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.equal.recursive.phpt [method=testDeep]
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.equal.recursive.phpt [method=testCross]
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.equal.recursive.phpt [method=testThirdParty]
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.error.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.exception.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.false.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.match.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.match.regexp.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.nan.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.null.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.same.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.true.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.truthy.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.type.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Collector.start.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/CommandLine.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/DataProvider.load.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/DataProvider.parseAnnotation.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/DataProvider.testQuery.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/DomQuery.css2Xpath.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/DomQuery.fromHtml.phpt
   Exited with error code 255 (expected 0)

-- FAILED: tester/tests/DomQuery.fromXml.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Dumper.dumpException.phpt
   Exited with error code 255 (expected 0)

-- FAILED: tester/tests/Dumper.toLine.phpt
   Exited with error code 255 (expected 0)

-- FAILED: tester/tests/Dumper.toPhp.phpt
   Exited with error code 255 (expected 0)

-- FAILED: tester/tests/Dumper.toPhp.recursion.phpt
   Exited with error code 255 (expected 0)

-- FAILED: tester/tests/FileMock.phpt
   Exited with error code 255 (expected 0)

-- FAILED: tester/tests/Helpers.escapeArg.phpt
   Exited with error code 255 (expected 0)

-- FAILED: tester/tests/Assert.contains.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/Assert.count.phpt
   Exited with error code 134 (expected 0)

-- FAILED: tester/tests/HhvmPhpInterpreter.phpt
   Exited with error code 255 (expected 0)
   E_WARNING: Invalid argument: stream_get_contents() expects parameter 1 to be a resource

   in Tester/Runner/HhvmPhpInterpreter.php(44)
   in Tester/Runner/HhvmPhpInterpreter.php(44) stream_get_contents()
   in tester/tests/bootstrap.php(21) Tester\Runner\HhvmPhpInterpreter->__construct()
   in tester/tests/HhvmPhpInterpreter.phpt(16) createInterpreter()

-- FAILED: tester/tests/Runner.find-tests.phpt
   Exited with error code 255 (expected 0)
   E_WARNING: Invalid argument: stream_get_contents() expects parameter 1 to be a resource

   in Tester/Runner/HhvmPhpInterpreter.php(44)
   in Tester/Runner/HhvmPhpInterpreter.php(44) stream_get_contents()
   in tester/tests/bootstrap.php(21) Tester\Runner\HhvmPhpInterpreter->__construct()
   in tester/tests/Runner.find-tests.phpt(14) createInterpreter()

-- FAILED: tester/tests/Runner.multiple.phpt
   Exited with error code 255 (expected 0)
   E_WARNING: Invalid argument: stream_get_contents() expects parameter 1 to be a resource

   in Tester/Runner/HhvmPhpInterpreter.php(44)
   in Tester/Runner/HhvmPhpInterpreter.php(44) stream_get_contents()
   in tester/tests/bootstrap.php(21) Tester\Runner\HhvmPhpInterpreter->__construct()
   in tester/tests/Runner.multiple.phpt(15) createInterpreter()

-- FAILED: tester/tests/Runner.annotations.phpt
   Exited with error code 255 (expected 0)
   E_WARNING: Invalid argument: stream_get_contents() expects parameter 1 to be a resource

   in Tester/Runner/HhvmPhpInterpreter.php(44)
   in Tester/Runner/HhvmPhpInterpreter.php(44) stream_get_contents()
   in tester/tests/bootstrap.php(21) Tester\Runner\HhvmPhpInterpreter->__construct()
   in tester/tests/Runner.annotations.phpt(29) createInterpreter()

-- FAILED: tester/tests/Runner.edge.phpt
   Exited with error code 255 (expected 0)
   E_WARNING: Invalid argument: stream_get_contents() expects parameter 1 to be a resource

   in Tester/Runner/HhvmPhpInterpreter.php(44)
   in Tester/Runner/HhvmPhpInterpreter.php(44) stream_get_contents()
   in tester/tests/bootstrap.php(21) Tester\Runner\HhvmPhpInterpreter->__construct()
   in tester/tests/Runner.edge.phpt(29) createInterpreter()

-- FAILED: tester/tests/Runner.multiple-fails.phpt
   Exited with error code 255 (expected 0)
   E_WARNING: Invalid argument: stream_get_contents() expects parameter 1 to be a resource

   in Tester/Runner/HhvmPhpInterpreter.php(44)
   in Tester/Runner/HhvmPhpInterpreter.php(44) stream_get_contents()
   in tester/tests/bootstrap.php(21) Tester\Runner\HhvmPhpInterpreter->__construct()
   in tester/tests/Runner.multiple-fails.phpt(30) createInterpreter()


FAILURES! (48 tests, 39 failures, 1 skipped, 14.2 seconds)
$ ./Tester/tester -p hhvm -j 1 tests/
 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v1.2.0

Note: No php.ini is used.
PHP 5.6.99-hhvm | 'hhvm' --php -d hhvm.log.always_log_unhandled_exceptions=false  | 1 thread

...................s...........................s


OK (48 tests, 2 skipped, 37.5 seconds)

Anyway, even HHVM's stable channel is totally unreliable, broken and buggy. And this is supposed to be LTS release. Sad.

@milo
Copy link
Member Author

milo commented Sep 22, 2014

@Majkl578 With the same version as you have, parallel run works for me fine. I don't know how to reproduce it.

@dg I think this is ready to be merged. It works with HHVM 3.3.0 and 3.4.0-dev+2014.09.21.

@dg
Copy link
Member

dg commented Sep 22, 2014

👍

milo added a commit that referenced this pull request Sep 22, 2014
HHVM fixes - tests pass hunting
@milo milo merged commit 31cedbb into nette:master Sep 22, 2014
@milo milo deleted the pull-hhvm-fixes branch September 22, 2014 13:51
@milo
Copy link
Member Author

milo commented Sep 22, 2014

@Majkl578 If you can debug HHVM parallel jobs problem, please, open a new issue.

@Majkl578
Copy link
Contributor

@milo: Strange. :) It's Debian Wheezy and HHVM 3.3.0 stable channel (their official repo). Not sure how reliably debug such thing remotely...

@Majkl578
Copy link
Contributor

@milo: Ok, I tried, but I give up. I don't know how to debug such thing. Tester is run by PHP 5.6.0 from Dotdeb, tests by HHVM 3.3.0. The problem is somewhere in Runner or Job. When I use e.g. 10 thread instead of default 33, it works. With 10, it is still ok, with 15, it starts to fail randomly (like 6 tests out of 48), the higher concurrency, the more random failures. Also, some tests died with the following error:

   Exited with error code 255 (expected 0)
   Exception: Unable to run '/usr/bin/hhvm': could not allocate 216006655 bytes for translation cache

   in Tester/Runner/HhvmPhpInterpreter.php(48)
   in tester/tests/bootstrap.php(21) Tester\Runner\HhvmPhpInterpreter->__construct()
   in tester/tests/Runner.multiple.phpt(15)

Edit: Checked free/top and realized, it might also be related to low memory, since HHVM is extremely expensive on RAM, so 33 threads may easily exhaust all free RAM.

@milo
Copy link
Member Author

milo commented Sep 22, 2014

@Majkl578 It really looks like free memory problem. HHVM consumes 6x more memory than PHP in average.

@Majkl578
Copy link
Contributor

Maybe we could set default threads count lower for HHVM? Like, say, 12? It's not only memory expensive, but also CPU expensive, way more than PHP.

@milo
Copy link
Member Author

milo commented Sep 22, 2014

At first, we need feedback from more than 2 people. My VPS and server are OK ;)

@vojta
Copy link

vojta commented Sep 22, 2014

I've just tested the lastest HHVM compiled on CentOS 7 and it looks OK:

# hhvm --version
HipHop VM 3.3.0 (rel)
Compiler: heads/HHVM-3.3-0-g32574d90c536f361dde4e66389b0ea6960809461
Repo schema: f04a880280324debd6d9c5640e1ddcee59c7e05f
Extension API: 20140829

# ./Tester/tester -p hhvm tests/
 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v1.2.0

Note: No php.ini is used.
PHP 5.6.99-hhvm | 'hhvm' --php -d hhvm.log.always_log_unhandled_exceptions=false  | 33 threads

...s.............................s..............


OK (48 tests, 2 skipped, 24.3 seconds)

@vojta
Copy link

vojta commented Sep 22, 2014

BTW, 33 jobs is not a good default for HHVM, it eats a lot of memory with no positive impact. The number of the system's CPU cores would fit much better.

@Majkl578
Copy link
Contributor

@vojta: 💯 Exactly what I said above. :)

@dg
Copy link
Member

dg commented Sep 22, 2014

@vojta crucial is, at which number of threads will be lowest elaped time

@vojta
Copy link

vojta commented Sep 22, 2014

@dg depends on the environment. In example, we are using a shared development server. A developer running a testcase with such concurrency completely stalls that machine for the other users. Another thing is swapping out some apps or wiping the VM cache.
But I understand your point when you run the tests on your (dedicated) notebook.

@dg
Copy link
Member

dg commented Sep 22, 2014

On my virtual Ubuntu (64bit, 4cores) are best results for 4…7 (on Tester testsuite).

@vojta
Copy link

vojta commented Sep 23, 2014

OK, "number of cpus" (+2) sounds reasonable to me - if the test case is CPU bound, not i/o bound. There's always a possibility that the test run can run out of memory, but is much lower than with parallel 33 jobs.

@Majkl578
Copy link
Contributor

@vojta: Sounds reasonable, although on VPS it will be often 1(+2) as there is usually one virtual cpu with a single thread.
But it's all about sane default value and I think it should not be set to something that kills everything else. Everyone is free to set e.g. 50 threads themselves using -j option. :)
(Also changing process' nice might be an interesting way.)

@Majkl578
Copy link
Contributor

@vojta, @dg: What about something like in #174?

@milo
Copy link
Member Author

milo commented Sep 23, 2014

Server with Intel Xeon X3360 (4 cores) Debian:
Tester's tests: 7 threads
GithubApi: about 20 threads
One of private lib: about 9

The speed inceasing is very small with threads > 8. And as I can see, it is similar on your side. So, let's say, default threads to 8?

@dg
Copy link
Member

dg commented Sep 23, 2014

I am fine with 8.

milo added a commit that referenced this pull request Sep 23, 2014
The reason is memory saving.
Based on #165 discussion.
@Majkl578
Copy link
Contributor

Majkl578 commented Oct 2, 2014

If anyone is interested, here is current status for Nette under HHVM. All components master + Tester 1.3.0@RC + HHVM 3.4.0-dev@gd54eed7 (including merged fix for facebook/hhvm#2684).
https://gist.github.com/Majkl578/e542712be9bd5b827fff

application/              FAILED 2x (FileResponse.contentDisposition.phpt, Presenter.paramChecking.phpt)
caching/                  FAILED 2x (FileStorage.call.phpt, FileStorage.wrap.phpt)
database/                 FAILED 1x [mysql, sqlite] (ResultSet.normalizeRow.mysql.phpt[mysql])
finder/                   PASSED
forms/                    PASSED
latte/                    PASSED
neon/                     PASSED
reflection/               FAILED 5x (Extension.phpt, GlobalFunction.phpt, ClassType.phpt, Parameter.defaultValues.phpt, Parameter.php53.phpt)
routing/                  FAILED 1x (Route.utf8Param.phpt)
security/                 PASSED
tokenizer/                PASSED
tracy/                    FAILED 15x (Debugger.E_ERROR.production.console.phpt, Debugger.E_COMPILE_ERROR.console.phpt, Debugger.E_ERROR.console.phpt, Debugger.E_RECOVERABLE_ERROR.console.phpt, Debugger.exception.production.console.phpt, Debugger.exception.nonhtml.phpt, Debugger.exceptionHandler.development.console.phpt, Debugger.exceptionHandler.production.console.phpt, Debugger.shut-up.error.phpt, Debugger.scream.phpt, Debugger.strict.console.phpt, Debugger.warnings.console.phpt, Dumper.toHtml().live.phpt, Dumper.toHtml().phpt)
bootstrap/                FAILED 2x (Configurator.developmentContainer.phpt, Configurator.minimalContainer.phpt)
component-model/          PASSED
di/                       FAILED 1x (IniAdapter.errors.phpt)
http/                     FAILED 5x (Session.handler.phpt, Session.start.error.phpt, Session.storage.phpt, SessionSection.remove.phpt, SessionSection.setExpiration().phpt)
mail/                     PASSED
php-generator/            FAILED 2x (Method.variadics.phpt, ClassType.from.trait.phpt)
robot-loader/             PASSED
safe-stream/              PASSED
tester/                   PASSED
utils/                    FAILED 5x (Arrays.grep().errors.phpt, Callback.invokeSafe.phpt, Image.factories.phpt, Strings.Regexp.errors.backtrack.phpt, Strings.Regexp.errors.compilation.phpt)

@JanJakes
Copy link

JanJakes commented Oct 2, 2014

@Majkl578 Could you paste somewhere a detailed dump of the failures?

@Majkl578
Copy link
Contributor

Majkl578 commented Oct 2, 2014

@JanJakes:
I made a tool to run tests for all components at once: https://gist.github.com/Majkl578/eafc31a469076a3e886b
Here is current status (a bit newer HHVM, rebuilt before running it): https://gist.github.com/Majkl578/8bf952b3b99d376981bf

@JanJakes
Copy link

JanJakes commented Oct 5, 2014

@Majkl578 Sweet, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants