An attempt to implemnt "preloading" ability. #3538

dstogov · 2018-09-18T19:37:22Z

On startup, PHP may execute a script defined by opcache.preload configuration directive.
All function and classes loaded by this script became permanent and available to all the following requests.
For example, it's possible to preload almost whole Zend Framework, and save significant time on each request.

This is an unfinished PoC yet.

dstogov · 2018-09-18T19:42:43Z

A preload script for Zend Framework

<?php
function _preload($preload, string $pattern = "/\.php$/", array $ignore = []) {
	if (is_array($preload)) {
		foreach ($preload as $path) {
			_preload($path, $pattern, $ignore);
		}
	} else if (is_string($preload)) {
		$path = $preload;
	    if (!in_array($path, $ignore)) {
			if (is_dir($path)) {
				if ($dh = opendir($path)) {
        			while (($file = readdir($dh)) !== false) {
        				if ($file !== "." && $file !== "..") {
		        		    _preload($path . "/" . $file, $pattern, $ignore);
						}
		    	    }
    		    	closedir($dh);
	    		}
			} else if (is_file($path) && preg_match($pattern, $path)) {
				if (!opcache_compile_file($path)) {
					trigger_error("Preloading Failed", E_USER_ERROR);
				}
			}
		}
	}
}

set_include_path(get_include_path() . PATH_SEPARATOR . realpath("/var/www/ZendFramework/library"));
_preload(["/var/www/ZendFramework/library"]);
?>

dstogov · 2018-09-18T19:49:45Z

@nikic, @laruence could you please take a look.

Actually, this is not exactly what I liked to do.
PHP still has to perform significant work after each request, restoring preloaded classes and functions into initial state, at accel_deactivate

May be you'll get some related ideas.

nikic · 2018-09-20T09:55:16Z

The main idea here is to save the overhead of class binding at run-time, is that right?

I'm wondering how invalidation would be handled for preloads. If one of the (potentially many) files included by the preload is changed, can we detect this efficiently and invalidate the necessary parts (or just everything)?

dstogov · 2018-09-20T11:01:29Z

@nikic

The main idea here is to save the overhead of class binding at run-time, is that right?

Not only. The idea is to completely eliminate compilation and opcache overhead (copying from SHM to process memory and insertions into function/class tables on each request). Using this technique, we might write standard functions and classes in PHP (similar to systemlib.php in HHVM).

I'm wondering how invalidation would be handled for preloads. If one of the (potentially many) files included by the preload is changed, can we detect this efficiently and invalidate the necessary parts (or just everything)?

No. The preloaded functions and classes must not be invalidated at all (like internal ones).

Persistent, linked classes would open a way for more aggressive optimizations, especially with conjunction with JIT. Unfortunately, the current implementation still have to copy zend_class_entry from SHM to process memory and this makes overhead and troubles.

Some related work was done in Java World: https://simonis.github.io/JBreak2018/CDS/cds.xhtm http://www.inf.usi.ch/faculty/nystrom/papers/cdn02-ecoop.pdf

beberlei · 2018-09-20T19:34:54Z

@dstogov this is amazing!

Could you explain how or if this would affect extensions that profile userland functions with regard to a.) hooking into zend_execute_ex or b.) overwrite the function pointers.

dstogov · 2018-09-21T06:02:58Z

Could you explain how or if this would affect extensions that profile userland functions with regard to a.) hooking into zend_execute_ex or b.) overwrite the function pointers.

The preloaded op_arrays is not very different from op_arrays restored from opcache. So profilers should work in the same way. However, preloaded op_arrays relive request boundary and should be reset into initial state.

alex-mashin · 2018-10-01T04:52:55Z

What would PHP code that checks whether preload feature is available and certain functions and classes are preloaded and defines them if they are not look like? Will it be necessary at all, or the code that tires to overload preloaded functions and classes will be ignored, so that even unmodified existing CMS's will benefit from this feature?
Will the opcache.preload option cause whole directories to be preloaded like the script for Zend framework that you posted above?

dstogov · 2018-10-01T10:05:08Z

Note, that currently this is just an idea and implementation is not good enough.

1. What would PHP code that checks whether preload feature is available and certain functions and classes are preloaded and defines them if they are not look like?

It's possible to check preloaded function and classes using function_exists/class exists, but these checks also may change the original logic of some apps (e.g Wordpress execute PHP scripts when some function not defined, but if the function is preloaded this code is not executed and application doesn't work as expected)

Will it be necessary at all, or the code that tires to overload preloaded functions and classes will be ignored, so that even unmodified existing CMS's will benefit from this feature?

Inclusion of preloaded scripts should work out of the box ("redefinitions" are going to be ignored), but redefinition in other scripts will trigger errors (Cannot redeclare ...)

2. Will the opcache.preload option cause whole directories to be preloaded like the script for Zend framework that you posted above?

opcache.prelaod accepts just a PHP script. This script may implement the actual loading in a way you need.

dstogov · 2018-10-04T15:48:23Z

@nikic see the next attempt based on immutable classes (first commit). Don't spend a lot of time, just give your attendance, and may be ideas about problems and improvement.

Now it makes 50 times speedup on ZF loading benchmark and ~35% on ZF1 HelloWorld (3700 req/sec vs 2700 req/sec). Although, the implementation is far from completion and there are few serious problems, now it looks mush more promising.

nikic · 2018-10-05T08:42:54Z

ext/opcache/ZendAccelerator.c

+				if (op_array->type == ZEND_USER_FUNCTION) {
+					ZEND_ASSERT(op_array->fn_flags & ZEND_ACC_IMMUTABLE);
+					if (op_array->static_variables) {
+						destroy_op_array(op_array);


Should this be freeing the op_array rather than just the static_variables table in the request_data?

For "immutable" arrays we have to free only static_variables. Everything else kept in SHM, except for run_time cache, that is in CG(arena). Or may be I misunderstood the question.

I mean, the code here currently frees the op_array, but the assertion above says that this is an immutable op_array, so I would expect it to only free static_variables.

nikic · 2018-10-05T08:50:26Z

ext/opcache/ZendAccelerator.c

@@ -2425,6 +2486,152 @@ int accel_post_deactivate(void)
 		return SUCCESS;
 	}

+	/* Free static variables of preloaded classes and functions */
+	if (ZCSG(preload_script)) {


The cleanup that is done here for immutable functions/classes, can all of it be skipped for fast shutdown? I.e. just zeroing the request data should be sufficient?

You must be right, but current implementation also may preload non-immutable classes, and we have to deal with them. I'll think how to solve this.

nikic · 2018-10-05T08:53:59Z

ext/opcache/ZendAccelerator.c

+				if (ce->num_interfaces) {
+					found = 1;
+					for (i = 0; i < ce->num_interfaces; i++) {
+						p = zend_hash_find_ptr(EG(class_table), ce->interface_names[i].lc_name);


Does EG(class_table) also include internal classes at this point? If so, I'm wondering how this will be handled on Windows where IIRC multiple processes (with internal classes at different addrs) may attach to one shm instance.

Yes. I know about this problem, but prefer to postpone it. e.g. don't support this feature on Windows or copy internal classes into SHM.

nikic · 2018-10-05T09:10:55Z

@dstogov I think the new approach looks reasonable. What I slightly dislike is that we now have two different code paths for the immutable and the non-immutable cases for runtime cache, static vars and static props. I'm wondering if it would make sense to make all functions/classes (post-link) immutable to make the way they are handled uniform. This would require making the request_data handling more dynamic (something similar to the object store: dynamic resizing + basic free list).

dstogov · 2018-10-05T11:16:20Z

@nikic thanks for review. Your thoughts about non-uniform handling make sense. I'll try to solve all implementation problems first, and then return to this question. May be replace index access through CG(request_data), by indirect pointer access (through special MAP region for pointers from SHM to process memory. This is what HotSpot does)

raziel057 · 2018-10-09T19:37:30Z

ext/opcache/ZendAccelerator.c

+#endif
+	if (!ZCG(mem)) {
+		zend_shared_alloc_unlock();
+		zend_accel_error(ACCEL_LOG_FATAL, "No enogh shared memory for preloading!");


No enough shared memory for preloading

@raziel057 No > Not

Then maybe
Not enough shared memory for preloading

😁

zbenc · 2018-10-10T11:58:19Z

@dstogov Thanks for working on this!
Would this enable static inlining of PHP functions within other preloaded files, e.g. helper functions from one file getting inlined at the call sites in other files?
That would be one more great benefit of preloading IMO.

dstogov · 2018-10-10T13:48:48Z

In general, immutable classes/functions and preloading would simplify various "Whole Program Optimization" methods, including class hierarchy analysis and inner script inlining. On the other hand, dynamic nature of PHP and defined by the language life-time of variables + destructors makes a lot of troubles. Currently, I think just about preloading, optimization is the future step.

KalleZ · 2018-10-11T01:20:21Z

ext/opcache/ZendAccelerator.c

@@ -84,6 +84,9 @@ typedef int gid_t;
 #include <immintrin.h>
 #endif

+// TODO: remove ???
+#include <sys/mman.h>


I know its a WIP but this part (along with the mmap call) won't work for Windows and it needs to be using the File Mapping API (hence the AppVeyor fail)

php-pulls · 2018-10-12T02:13:10Z

Comment on behalf of petk at php.net:

Labelling

dstogov · 2018-10-12T10:26:55Z

@nikic please take a quick look into the next iteration. It's still not finished, but now immutable classes are supported in ZTS build and on Windows. I non-ZTS build the patch makes improvement even without preloading, because immutable classes are not copied into process memory on each request.
I hope, I have to solve only few edge cases (file_cache, opcache restart, preloading in ZTS and Windows). We will also need new API instead of run-time op_array->reserved[] access of immutable functions.(I'm going to reserve space in op_array->run_time_cache).
Do you see any other problems?

nikic

I looked only at the Zend part (not opcache). I think the approach makes sense and I don't see any immediate problems.

I think it would make sense to separate out the introduction of the map ptr mechanism and all the related changes and land them before the preloading functionality, as they are mostly independent and preloading will probably need more work before it's practically usable.

nikic · 2018-10-12T11:07:50Z

Zend/zend.c

+
+	if (CG(map_ptr_last) >= CG(map_ptr_size)) {
+#if ZEND_MAP_PTR_KIND == ZEND_MAP_PTR_KIND_PTR
+		// TODO: error ???


I'm not totally sure about the significance of the initial buffer size. Is this effectively the limit on (approximately) the number of functions that can be preloaded by opcache (but not how many can be loaded at runtime, as they don't go through this map)?

Right. This is how many pointers from opcache SHM to PROCESS we may have.

nikic · 2018-10-12T11:16:40Z

Zend/zend_API.c

-			}
+			ZEND_MAP_PTR_NEW(ce->static_members_table);
+#else
+			ZEND_MAP_PTR_INIT(ce->static_members_table, pecalloc(1, sizeof(void*), 1));


Why is the ZTS/NTS distinction here necessary?

With NTS we may avoid MAP slot allocation, but, probably, you are right, and it makes sense to try unifying this.

nikic · 2018-10-12T12:58:02Z

Zend/zend_interfaces.c

+
+static zend_always_inline zend_class_iterator_funcs* zend_get_iterator_funcs_ptr(zend_class_entry *ce) /* {{{ */
+{
+	if (ZEND_MAP_PTR_GET(ce->iterator_funcs_ptr)) {


Shouldn't this use ZEND_MAP_PTR?

No. Immutable classes are going to have ZEND_MAP_PTR set (pointer to the MAP region), but initially ZEND_MAP_PTR_GET is going to be NULL.

nikic · 2018-10-12T13:36:04Z

Zend/zend_closures.c

 			my_function.op_array.fn_flags |= ZEND_ACC_HEAP_RT_CACHE;
-			memset(my_function.op_array.run_time_cache, 0, my_function.op_array.cache_size);
+			ptr = emalloc(sizeof(void*) + my_function.op_array.cache_size);


It may be worthwhile to add a helper for this type of allocation (for request and arena), which has an additional prefixed indirection pointer. It's used a couple of times here and also for the iterator funcs.

dstogov · 2018-10-12T15:46:43Z

@nikic thanks for review. I completely agree, about separation of immutable+map_ptr and preloading. This is what I tried to do in first two commits. I hope, I'll able to deliver first part on next week.

pmmaga · 2018-11-08T11:01:56Z

Hi, currently, trying to include a composer generated autoloader in the preload script is failing an assertion (debug maintainer-zts):

[Thu Nov  8 10:54:40 2018]  Script:  '-'
/home/pedro/dev/php/php-src/Zend/zend_constants.c(477) :  Freeing 0x00007f92cfc03038 (24 bytes), script=-
Last leak repeated 2 times
[Thu Nov  8 10:54:40 2018]  Script:  '-'
/home/pedro/dev/php/php-src/Zend/zend_objects.c(180) :  Freeing 0x00007f92cfc0b600 (200 bytes), script=-
[Thu Nov  8 10:54:40 2018]  Script:  '-'
/home/pedro/dev/php/php-src/Zend/zend_string.h(132) :  Freeing 0x00007f92cfcbc1e0 (40 bytes), script=-
Last leak repeated 2 times
=== Total 7 memory leaks detected ===
php: /home/pedro/dev/php/php-src/Zend/zend_constants.c:61: copy_zend_constant: Assertion `(((c)->value).u2.constant_flags & 0xff) & (1<<1)' failed.
Aborted

This is the autoloader generated by cloning the latest master of composer, running composer install and then trying to include the vendor/autoload.php.
FWIW, a simple autoloader works:

<?php

spl_autoload_register(function($name) {
    include_once("$name.php");
});

use Foo;

Let me know if you need any additional info.

dstogov · 2018-11-08T11:37:27Z

composer/composer#7777 (comment)

Another question to clarify memory consumption

I also mean it works that way, persistent class opcodes does not copied into process memory from SHM, but shared between fpm processes

Could you confirm that, please?

Right.

dstogov · 2018-11-08T11:38:51Z

OK, so using include can be safe

I tried out php.ini's auto_prepend_file directive

auto_prepend_file = pre.php

bool(true)
bool(true)
bool(true)
bool(true)

So persistent classes available in auto_prepend_file too.

It makes sense to prohibit auto_prepend/append for preloading. I'll fix this.

friizer · 2018-11-08T12:05:06Z

Is there a problem with that, or why do you want to prohibit?

I dont have a problem with it, just checked to make sure it works too

dstogov · 2018-11-08T12:16:04Z

ops. Everything is fine. I thought prepend file was preloaded, but this is not true.

dstogov · 2018-11-08T12:30:44Z

Hi, currently, trying to include a composer generated autoloader in the preload script is failing an assertion (debug maintainer-zts):
[Thu Nov  8 10:54:40 2018]  Script:  '-'
/home/pedro/dev/php/php-src/Zend/zend_constants.c(477) :  Freeing 0x00007f92cfc03038 (24 bytes), script=-
Last leak repeated 2 times
[Thu Nov  8 10:54:40 2018]  Script:  '-'
/home/pedro/dev/php/php-src/Zend/zend_objects.c(180) :  Freeing 0x00007f92cfc0b600 (200 bytes), script=-
[Thu Nov  8 10:54:40 2018]  Script:  '-'
/home/pedro/dev/php/php-src/Zend/zend_string.h(132) :  Freeing 0x00007f92cfcbc1e0 (40 bytes), script=-
Last leak repeated 2 times
=== Total 7 memory leaks detected ===
php: /home/pedro/dev/php/php-src/Zend/zend_constants.c:61: copy_zend_constant: Assertion `(((c)->value).u2.constant_flags & 0xff) & (1<<1)' failed.
Aborted
This is the autoloader generated by cloning the latest master of composer, running composer install and then trying to include the vendor/autoload.php.
FWIW, a simple autoloader works:
<?php

spl_autoload_register(function($name) {
    include_once("$name.php");
});

use Foo;
Let me know if you need any additional info.

I hope this should be fixed now.

Tobion · 2018-11-08T19:17:11Z

I assume the preloading does not have any affect in CLI?
Do these settings affect preloading in any way, esp. file_cache_only? Would this disable preloading because there is no shared memory to put the preloaded stuff into? Or is preload not executed in CLI anyway?

opcache.preload = myscript.php
opcache.enable_cli = 1
opcache.file_cache = "/dev/shm/php-opcache"
opcache.file_cache_only = 1

dstogov · 2018-11-09T07:52:40Z

I assume the preloading does not have any affect in CLI?
Do these settings affect preloading in any way, esp. file_cache_only? Would this disable preloading because there is no shared memory to put the preloaded stuff into? Or is preload not executed in CLI anyway?
opcache.preload = myscript.php
opcache.enable_cli = 1
opcache.file_cache = "/dev/shm/php-opcache"
opcache.file_cache_only = 1

Preloading works in CLI, as well, but doesn't work without opcache or in file_cache_only mode.

…ernal ones.

friizer · 2018-11-09T15:39:11Z

Thanks,

That fixed last of my errors with preloading

I just run my functional tests against the API server (symfony stack),
Nearly 1000 tests run over one hour, through the full stack, without a single crash

Over 10000 http requests, aprx 4000 transactions, 60000 db queries worked without single problem

Tobion · 2018-11-09T22:38:01Z

Preloading works in CLI, as well, but doesn't work without opcache or in file_cache_only mode.

@dstogov what does it mean it works? Does it mean, the preloading is executed and does not fail? But it does not give any performance advantage because every CLI execution would do it's own preloading that cannot be shared?

dstogov · 2018-11-13T12:54:15Z

@Tobion preloading doesn't improve performance in CLI, but anyway, it loads file(s) before the main script and makes all the preloaded entities available for this script.

dstogov · 2018-11-14T13:46:55Z

merged into master

jamespatterson · 2018-11-14T16:03:10Z

I am currently using a custom PHP implementation for process management, primarily to get the advantage of preloaded code and config data, but also to communicate with worker processes (forks of the master process) using IPC to direct config reloads, preloaded resource reloads, etc. This sounds like I could use FPM for all the preloading, as long as I stop depending on a long-lived process that acts as the controller. Is there any way with FPM to load custom PHP code into the process manager? Or is all this preloading done per-process spawn instead of once per FPM startup?

dstogov · 2018-11-14T20:02:35Z

@jamespatterson preloading is done once per main PHP process and available for all forked workers. It helps keeping code between request, but it doesn't keep the state (variables, config data, constructed arrays/object graphs etc).

csdougliss · 2018-11-15T11:11:35Z

Is there a way to clear the pre-load similar to opcache_clear(); ?

For example, if you deploy a new set of code to your existing running system (I'm thinking Jenkins, to running web server, with changed php code?)

dstogov · 2018-11-15T14:04:31Z

@ craigcarnell no, you'll have to restart PHP.

kocoten1992 · 2019-01-19T10:51:06Z

Hi @dstogov, is this feature work well with service reload ?
https://github.com/php/php-src/blob/master/sapi/fpm/php-fpm.service.in#L13

nikic · 2019-01-19T11:31:44Z

@kocoten1992 No, this feature is not compatible with reloads. It requires a full php-fpm restart.

garygreen · 2019-04-24T11:50:17Z

Preloading Limitation

Only classes without unresolved parent, interfaces, traits and constant values may be preloaded.

Can anyone explain what it means by "unresolved parent"?

dstogov · 2019-04-24T12:25:22Z

class A extends B {
}

if class with name "B" is not found in preloaded scripts, then A has an "unresolved parent".

Rookiebrother · 2019-05-19T00:36:46Z

How should I call
final class A implements Serializable{
Public function __construct()
{
Throw new \Exception('No implemented');
}
}
Thank you

Tobion · 2019-05-19T08:53:51Z

@LittleRookie2015 class_exists(A::class);

brendt · 2019-06-27T13:36:30Z

I'm sorry if this is not the right place to ask this question, but I couldn't seem to find anything written about this in the RFC.

When exactly will a class be "unlinked"? https://github.com/php/php-src/pull/3538/files#diff-f6d4b551c6233faa294ae9a97897c7d4R3368

dstogov · 2019-06-27T13:42:41Z

@brendt
Preloading tries to combine classes from different files and link them together (perform inheritance etc), but this is not always possible, because, some classes may be missed (or compiled conditionally). So, if for some class we didn't find parent class (or some of interfaces or traits) we can't link it.

brendt · 2019-06-27T13:56:48Z

@dstogov so I'm trying to preload the whole Laravel framework as a test, and getting thousands of these warnings. What's confuses me though, is that when trying to preload a single file, it works.

Here's an example, this is my preload script, it's located in the project root dir.

<?php

class Preloader
{
    private array $paths;

    public function __construct(string ...$paths)
    {
        $this->paths = $paths;
    }

    public function load(): void
    {
        foreach ($this->paths as $path) {
            $this->loadPath($path);

            set_include_path($path);
        }
    }

    protected function loadPath(string $path): void
    {
        if (is_dir($path)) {
            $this->loadDir($path);

            return;
        }

        if (substr($path, -4) !== '.php') {
            return;
        }

        $isPreloaded = opcache_compile_file($path);

        if (! $isPreloaded) {
            trigger_error("Preloading failed", E_USER_ERROR);
        }
    }

    protected function loadDir(string $path): void
    {
        $handle = opendir($path);

        while ($file = readdir($handle)) {
            if (in_array($file, ['.', '..'])) {
                continue;
            }

            $this->loadPath($path . "/" . $file);
        }

        closedir($handle);
    }
}

(new Preloader([
    __DIR__ . '/vendor/laravel',
]))->load();

this script shows thousands of warnings like this one:

php_1    | <b>Warning</b>:  Can't preload unlinked class Illuminate\Database\Eloquent\Relations\HasMany in <b>/shared/httpd/preload/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Relations/HasMany.php</b> on line <b>7</b><br />
php_1    | <br />

However loading, for example, /shared/httpd/preload/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Relations/HasMany.php on its own, works:

(new Preloader(
    __DIR__ . '/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Relations/HasMany.php',
))->load();

brendt · 2019-06-28T08:26:42Z

A few hours later and I get what's happening: all classes used by a class must be preloaded as well, in order for that class to be prelaodable.

dstogov force-pushed the preload branch from 9f6de1b to cf819ee Compare October 4, 2018 15:30

nikic reviewed Oct 5, 2018

View reviewed changes

raziel057 reviewed Oct 9, 2018

View reviewed changes

KalleZ reviewed Oct 11, 2018

View reviewed changes

php-pulls added the Feature label Oct 12, 2018

nikic reviewed Oct 12, 2018

View reviewed changes

dstogov force-pushed the preload branch from 59c96eb to e584df0 Compare October 15, 2018 18:51

dstogov added 2 commits October 15, 2018 23:34

Immutable clases and op_arrays

9409958

An attempt to implemnt "preloading" ability.

0810ce0

dstogov force-pushed the preload branch from e584df0 to 0810ce0 Compare October 15, 2018 20:40

dstogov mentioned this pull request Oct 15, 2018

Immutable clases and op_arrays #3608

Closed

Fixed incorrect signal handlers overriding

f76a955

Don't preload constants defined during preload script excution.

34645ae

Properly resolve magic method of preloaded classes inherited from int…

45fdd03

…ernal ones.

dstogov closed this Nov 14, 2018

hussainweb mentioned this pull request Jul 13, 2020

The class_alias in bootstrap.php causes warnings when using preloading Chi-teck/drupal-code-generator#47

Closed

An attempt to implemnt "preloading" ability. #3538

An attempt to implemnt "preloading" ability. #3538

Conversation

dstogov commented Sep 18, 2018

dstogov commented Sep 18, 2018

dstogov commented Sep 18, 2018

nikic commented Sep 20, 2018

dstogov commented Sep 20, 2018

beberlei commented Sep 20, 2018

dstogov commented Sep 21, 2018

alex-mashin commented Oct 1, 2018

dstogov commented Oct 1, 2018

dstogov commented Oct 4, 2018

nikic Oct 5, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikic commented Oct 5, 2018

dstogov commented Oct 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zbenc commented Oct 10, 2018

dstogov commented Oct 10, 2018

Choose a reason for hiding this comment

php-pulls commented Oct 12, 2018

dstogov commented Oct 12, 2018

nikic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dstogov commented Oct 12, 2018

pmmaga commented Nov 8, 2018

dstogov commented Nov 8, 2018

dstogov commented Nov 8, 2018

friizer commented Nov 8, 2018

dstogov commented Nov 8, 2018

dstogov commented Nov 8, 2018

Tobion commented Nov 8, 2018 • edited

dstogov commented Nov 9, 2018

friizer commented Nov 9, 2018

Tobion commented Nov 9, 2018

dstogov commented Nov 13, 2018

dstogov commented Nov 14, 2018

jamespatterson commented Nov 14, 2018

dstogov commented Nov 14, 2018

csdougliss commented Nov 15, 2018 • edited

dstogov commented Nov 15, 2018

kocoten1992 commented Jan 19, 2019

nikic commented Jan 19, 2019

garygreen commented Apr 24, 2019 • edited

Preloading Limitation

dstogov commented Apr 24, 2019

Rookiebrother commented May 19, 2019

Tobion commented May 19, 2019

brendt commented Jun 27, 2019

dstogov commented Jun 27, 2019

brendt commented Jun 27, 2019

brendt commented Jun 28, 2019

nikic Oct 5, 2018 •

edited

Tobion commented Nov 8, 2018 •

edited

csdougliss commented Nov 15, 2018 •

edited

garygreen commented Apr 24, 2019 •

edited