New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hard to reproduce cache issue with one of several symptoms [bounty馃挵] #2007

Closed
jasonvarga opened this Issue Jun 6, 2018 · 58 comments

Comments

Projects
None yet
@jasonvarga
Member

jasonvarga commented Jun 6, 2018

Describe the bug

The cache will randomly appear to have lost some content.
(Just the cache, though, the data in the files is fine.)

Users may see one or more of the following symptoms, all of which can usually be remedied by clearing the cache:

  • Users get logged out (and may or may not be able to log back in)
  • One, multiple, or all pages result in 404s.
  • The relate tag returns no results, or the get modifier outputs an ID.
  • Other things that expect content to be returned may result in an error.

To Reproduce

So far, it's not reliably reproducible.

We believe it is caused by a race condition. Something along the lines of while the cache is being updated in one request, another request loads (and possibly re-writes) a partly-written cache.

To temporarily fix

Clear the cache to get things working again.

php please cache:clear or remove the contents of local/storage/framework/cache

Additional context

Some users have reduced the frequency of this by doing one or more of the following:

  • Setting SESSION_DRIVER=cookie in their .env file.
  • Putting the cache in Redis

Related issues:

In this comment, I think @jackwakefield is on the right track. We probably need to add locking while the stache is being updated that prevents other requests updating at the same time. The problem is we don't want to implement something without being able to reproduce the bug first.

Bounty 馃挵馃挵馃挵

The first person to reliably and consistently recreate this issue leading to a fix, will be the proud new owner of a $250 Amazon gift card.

Bounty Hunter Boba Fett

@jasonvarga jasonvarga added the bug label Jun 6, 2018

@jackmcdade jackmcdade changed the title from Hard to reproduce cache issue with one of several symptoms [with bounty!] to Hard to reproduce cache issue with one of several symptoms [bounty馃挵] Jun 7, 2018

@anthubbard

This comment has been minimized.

anthubbard commented Jun 7, 2018

What version do we need to be on to reproduce it? I get this bug all the time and have done since v2.1

@fkwakkenbos

This comment has been minimized.

fkwakkenbos commented Jun 7, 2018

We've experienced this issue while a user is editing content within the CP and at the same time a (automatic) deployment is clearing the cache by running php please cache:clear.
When the timing is correct, this causes the cache to be invalid, which results in returning empty content Models or Collections within the Statamic Core.
Since we've disabled the auto-deploy and make sure nobody is editing within the CP while deploying, this issue did never occurred again!

@FrittenKeeZ

This comment has been minimized.

FrittenKeeZ commented Jun 7, 2018

One way to reproduce a race condition could be a webserver stress test, using fx. Siege (latest version is 4.0.4) to hammer your site repeatedly while saving stuff in CP.
We have a client who's asking us for an ETA on when this issue will be fixed, so any information on progress would be very appreciated.

@jackmcdade

This comment has been minimized.

Member

jackmcdade commented Jun 7, 2018

We've tried these things with no luck. I know they can trigger it, but so far have never been able to for us. Haven't tried Siege specifically though, worth a shot. We need to be able to reproduce with 100% consistency in order to work on a solution.

@FrittenKeeZ

This comment has been minimized.

FrittenKeeZ commented Jun 7, 2018

Have you tried creating a unit test using Guzzle to send an async request forcing a stache/cache update, and at the same time multiple async read request? Might be easier and more consistent than using Siege.

@jackwakefield

This comment has been minimized.

jackwakefield commented Jun 7, 2018

This isn't an attempt to claim the bounty, and I have no reliable method of reproducing this, but the following patch is a slightly updated version of the one linked above. With this, as well as using opcache as the cache driver (using a custom version of laravel-opcache which works with Statamic/Laravel 5.1), I've not hit this issue in 1-2 months.

Note: this does differ slightly from the standard behaviour, by blocking simultaneous API requests until the Stache has been rebuilt.

diff --git a/statamic/core/Providers/StacheServiceProvider.php b/statamic/core/Providers/StacheServiceProvider.php
index 6819d6f..b31bb7f 100644
--- a/statamic/core/Providers/StacheServiceProvider.php
+++ b/statamic/core/Providers/StacheServiceProvider.php
@@ -100,7 +100,12 @@ class StacheServiceProvider extends ServiceProvider
 
         $this->app->make(Stache::class)->locales(Config::getLocales());
 
-        $this->handleColdCache();
+        // On large sites, the Stache may take some time to build initially. If another request
+        // hits Statamic while it's in the middle of being built, it may use a half-created
+        // cache resulting in missing data. Here, we'll exit early with a simple refresh
+        // meta tag. Once the Stache is built, the page will resume loading as usual.
+        while ($this->stache->isPerformingInitialWarmUp()) {
+        }
 
         $this->manager = $this->app->make(Manager::class);
 
@@ -116,6 +121,12 @@ class StacheServiceProvider extends ServiceProvider
         // we should update on each request, or whether it's a glide route.
         $update = $this->shouldUpdateStache();
 
+        $this->stache->lock();
+
+        register_shutdown_function(function () {
+            $this->stache->unlock();
+        });
+
         try {
             // At this point the Stache is just an empty object. We'll want to load
             // (aka. 'hydrate') it. We'll also mark it as warmed. If it turns
@@ -136,6 +147,7 @@ class StacheServiceProvider extends ServiceProvider
             $this->updateStache();
         }
 
+        $this->stache->unlock();
         $this->stache->heat();
     }
 
@@ -153,9 +165,7 @@ class StacheServiceProvider extends ServiceProvider
         try {
             $this->manager->update();
         } catch (Exception $e) {
-            if (File::exists($this->stache->building_path)) {
-                File::delete($this->stache->building_path);
-            }
+            $this->stache->unlock();
 
             throw $e;
         }
@@ -205,33 +215,6 @@ class StacheServiceProvider extends ServiceProvider
         return [Stache::class];
     }
 
-    /**
-     * Handle the cold cache
-     *
-     * On large sites, the Stache may take some time to build initially. If another request
-     * hits Statamic while it's in the middle of being built, it may use a half-created
-     * cache resulting in missing data. Here, we'll exit early with a simple refresh
-     * meta tag. Once the Stache is built, the page will resume loading as usual.
-     *
-     * @return void
-     */
-    private function handleColdCache()
-    {
-        if (app()->runningInConsole()) {
-            return;
-        }
-
-        $start = time();
-
-        while ($this->stache->isPerformingInitialWarmUp()) {
-            if (time() - $start >= 10) {
-                $this->outputRefreshResponse();
-            }
-
-            sleep(1);
-        }
-    }
-
     /**
      * When the Stache is being built, we'll output a refresh/redirect until it's ready.
      *
@@ -239,31 +222,12 @@ class StacheServiceProvider extends ServiceProvider
      */
     private function outputRefreshResponse()
     {
-        if ($this->isAjaxRequest()) {
-            http_response_code(503);
-            exit(t('stache_building'));
-        }
-
         $html = sprintf('<meta http-equiv="refresh" content="1; URL=\'%s\'" />', request()->getUri());
 
         exit($html);
     }
 
-    private function isAjaxRequest()
-    {
-        return request()->ajax() || request()->wantsJson();
-    }
-
     private function cleanUpForConsole()
     {
-        if (! app()->runningInConsole()) {
-            return;
-        }
-
-        if (File::exists($lock = $this->stache->building_path)) {
-            File::delete($lock);
-        }
-
-        Config::set('system.ensure_unique_ids', false);
     }
 }
diff --git a/statamic/core/Stache/Stache.php b/statamic/core/Stache/Stache.php
index 841dbbc..b75ae99 100644
--- a/statamic/core/Stache/Stache.php
+++ b/statamic/core/Stache/Stache.php
@@ -69,6 +69,8 @@ class Stache
 
     public $taxonomies;
 
+    protected $locked;
+
     /**
      * Stache constructor.
      *
@@ -86,6 +88,28 @@ class Stache
         $this->temperature = self::TEMP_COLD;
     }
 
+    public function lock()
+    {
+        if (!$this->locked) {
+            $handle = fopen($this->building_path, 'c');
+            flock($handle, LOCK_EX);
+            fclose($handle);
+
+            $this->locked = true;
+        }
+    }
+
+    public function unlock()
+    {
+        if ($this->locked) {
+            $handle = fopen($this->building_path, 'c');
+            flock($handle, LOCK_UN);
+            fclose($handle);
+
+            $this->locked = false;
+        }
+    }
+
     /**
      * Change the temperature to cold
      *
@@ -94,10 +118,6 @@ class Stache
     public function cool()
     {
         $this->temperature = self::TEMP_COLD;
-
-        if (! File::exists($this->building_path)) {
-            File::put($this->building_path, true);
-        }
     }
 
     /**
@@ -108,10 +128,6 @@ class Stache
     public function heat()
     {
         $this->temperature = self::TEMP_WARM;
-
-        if (File::exists($this->building_path)) {
-            File::delete($this->building_path);
-        }
     }
 
     /**
@@ -141,7 +157,11 @@ class Stache
      */
     public function isPerformingInitialWarmUp()
     {
-        return File::exists($this->building_path);
+        $handle = fopen($this->building_path, 'c');
+        $locked = !flock($handle, LOCK_EX | LOCK_NB);
+        fclose($handle);
+
+        return $locked;
     }
 
     /**
@jackmcdade

This comment has been minimized.

Member

jackmcdade commented Jun 7, 2018

Thanks @jackwakefield! Our hope is to find a 100% repeatable/testable way to recreate the issue, then apply your patch, and see what happens :)

@jasonvarga

This comment has been minimized.

Member

jasonvarga commented Jun 7, 2018

@jackwakefield Why'd you replace the handleColdCache method with a basic while loop? Is that just your patch being re-applied over the top of an update we've since made?

@jackwakefield

This comment has been minimized.

jackwakefield commented Jun 7, 2018

@jasonvarga Yep exactly that, we also needed slightly different behaviour for deadlocks where the script would be killed "naturally" when exceeding max_execution_time (rather than the 503 response or meta refresh), but thankfully this hasn't happened yet.

@Keoghan

This comment has been minimized.

Keoghan commented Jun 8, 2018

Unfortunately I don't quite know my way round well enough to give this in a fully formed way, sorry. But for testing purposes, how about replacing the Local File System Adapter (specifically for Stache, if possible) with one that is deliberately slowed down? Then when firing in a request as the Stache is slowly updating, hopefully it simulates the race.

e.g. SlowLocalAdapter

<?php

namespace League\Flysystem\Adapter;

use League\Flysystem\Config;

class SlowLocal extends Local
{
    protected $sleep = 10000;

    public function setSleep($microSeconds = 0)
    {
        $this->sleep = $microSeconds;
    }

    /**
     * @inheritdoc
     */
    public function has($path)
    {
        usleep($this->sleep);

        return parent::has($path);
    }

    /**
     * @inheritdoc
     */
    public function write($path, $contents, Config $config)
    {
        usleep($this->sleep);

        return parent::write($path, $contents, $config);
    }

    /**
     * @inheritdoc
     */
    public function writeStream($path, $resource, Config $config)
    {
        usleep($this->sleep);

        return parent::writeStream($path, $contents, $config);
    }

    /**
     * @inheritdoc
     */
    public function readStream($path)
    {
        usleep($this->sleep);

        return parent::readStream($path);
    }

    /**
     * @inheritdoc
     */
    public function updateStream($path, $resource, Config $config)
    {
        usleep($this->sleep);

        return parent::updateStream($path, $resource, $config);
    }

    /**
     * @inheritdoc
     */
    public function update($path, $contents, Config $config)
    {
        usleep($this->sleep);

        return parent::update($path, $resource, $config);
    }

    /**
     * @inheritdoc
     */
    public function read($path)
    {
        usleep($this->sleep);

        return parent::read($path);
    }

    /**
     * @inheritdoc
     */
    public function rename($path, $newpath)
    {
        usleep($this->sleep);

        return parent::rename($path, $newpath);
    }

    /**
     * @inheritdoc
     */
    public function copy($path, $newpath)
    {
        usleep($this->sleep);

        return parent::copy($path, $newpath);
    }

    /**
     * @inheritdoc
     */
    public function delete($path)
    {
        usleep($this->sleep);

        return parent::delete($path);
    }

    /**
     * @inheritdoc
     */
    public function listContents($directory = '', $recursive = false)
    {
        usleep($this->sleep);

        return parent::listContents($directory, $recursive);
    }

    /**
     * @inheritdoc
     */
    public function getMetadata($path)
    {
        usleep($this->sleep);

        return parent::getMetadata($path);
    }

    /**
     * @inheritdoc
     */
    public function getSize($path)
    {
        usleep($this->sleep);

        return parent::getSize($path);
    }

    /**
     * @inheritdoc
     */
    public function getMimetype($path)
    {
        usleep($this->sleep);

        return parent::getMimetype($path);
    }

    /**
     * @inheritdoc
     */
    public function getTimestamp($path)
    {
        usleep($this->sleep);

        return parent::getTimestamp($path);
    }

    /**
     * @inheritdoc
     */
    public function getVisibility($path)
    {
        usleep($this->sleep);

        return parent::getVisibility($path);
    }

    /**
     * @inheritdoc
     */
    public function setVisibility($path, $visibility)
    {
        usleep($this->sleep);

        return parent::setVisibility($path, $visibility);
    }

    /**
     * @inheritdoc
     */
    public function createDir($dirname, Config $config)
    {
        usleep($this->sleep);

        return parent::createDir($dirname, $config);
    }

    /**
     * @inheritdoc
     */
    public function deleteDir($dirname)
    {
        usleep($this->sleep);

        return parent::deleteDir($dirname);
    }
}
@jasonvarga

This comment has been minimized.

Member

jasonvarga commented Jun 9, 2018

@jackwakefield In your snippet, your lock method looks like this:

+    public function lock()
+    {
+        if (!$this->locked) {
+            $handle = fopen($this->building_path, 'c');
+            flock($handle, LOCK_EX);
+            fclose($handle);
+
+            $this->locked = true;
+        }
+    }

You obtain the lock and then immediately fclose it. Doesn't fclose also release the lock?

@jackwakefield

This comment has been minimized.

jackwakefield commented Jun 9, 2018

@jasonvarga PHP 5.3.2 changed that behaviour:

https://secure.php.net/manual/en/function.flock.php

Changelog
5.3.2 | The automatic unlocking when the file's resource handle is closed was removed. Unlocking now always has to be done manually.

@steveooo9

This comment has been minimized.

steveooo9 commented Jun 10, 2018

Hmm. Maybe some issue with the server memory limits? On some other project, I had some random issues (things not displaying without errors) and by increasing the memory limit's things worked. Maybe somewhere things get loaded into memory and don't throw errors when memory is full? Just my 2 cents about this issue

@jasonvarga jasonvarga referenced this issue Jun 11, 2018

Open

Discussion #1

@steveooo9

This comment has been minimized.

steveooo9 commented Jun 11, 2018

I have no idea if this might help, but now another idea:
I use file_exists for checking a file and I found that in my local env somehow, file_exists and fopen and such weren't updated instantly. This resolved when moving my code to another server.

I used file_exists on another project to check if a custom lock file is there to lock out users from the admin panel. I tested locally and had the issue that I added the lockfile that was checked by file_exists but it wouldn't return true. Doing the same thing (removing file and adding file) and checking if file_exists is instantly, I found out that locally it took some time to return true whereas on our live servers, it worked instantly.

Also I just added clearstatcache(); but don't know exactly why I still have this in my code. Maybe needed?

@FrittenKeeZ

This comment has been minimized.

FrittenKeeZ commented Jun 18, 2018

Do you have any status regarding fixing this issue?
We have a client frequently inquiring when a permanent solution is available.

@jackmcdade

This comment has been minimized.

Member

jackmcdade commented Jun 18, 2018

@steveooo9

This comment has been minimized.

steveooo9 commented Jun 20, 2018

Would be great if you could share the issue. Very interesting and everyone knows those hard bugs everyone hates.

@Madsbie

This comment has been minimized.

Madsbie commented Jun 25, 2018

Hi, we are experiencing this issue frequently, which is leaving our client with a site that breaks on more or less a weekly basis. We've implemented a failsafe that cleares the cache in event of errors, leaving the site down for only a few minutes - when the failsafe works.
Can you let us know when a permanent solution will be in place?

@jackmcdade

This comment has been minimized.

Member

jackmcdade commented Jun 25, 2018

@Madsbie I wish I could. It's in the works, and once we're confident in it, we'll roll it out and update this thread.

@Madsbie

This comment has been minimized.

Madsbie commented Jun 27, 2018

Hey @jackmcdade i understand that chasing down a race-condition is complicated, and i'm sure it wont be long before a permanent fix is in production.

In the meantime I have begun implementing Statamic as our preferred CMS when working with clients in need of "less-than-enterprise" solutions - they are starting to worry and question that decision.

Could you possibly advice as to when you can ensure an update and priority as to progress? I would feel much more confident being able to communicate on the back of a statement coming from you.

best regards.

@jackmcdade

This comment has been minimized.

Member

jackmcdade commented Jun 27, 2018

Yes of course. It's still in the works, and we're working on it, and when there's an update, it'll be posted here. Hang in there!

@jasonvarga

This comment has been minimized.

Member

jasonvarga commented Jun 29, 2018

I have a WIP that I'd be happy to share with someone who is willing to test it out.

Please email us at support@statamic.com saying you want to try it out, and give us a little description of exactly what your issue is (since this is fairly general and has different symptoms for different people)

@patrikkernke

This comment has been minimized.

patrikkernke commented Jul 5, 2018

@jasonvarga I have called out for ya through email (support@statmic.com). I would gladly test the WIP. Nothing what I do changes the circumstance, that my client can鈥榯 log in in the CP (鈥濻ession expired鈥). But only in the production-environment.

Thanks @jasonvarga I found a mistake I had done :) It has nothing to do with this issue!

@patrikkernke

This comment has been minimized.

patrikkernke commented Jul 5, 2018

Thanks @jasonvarga I found a mistake I have made :) It has nothing to do with this issue!

@RomainFrancony

This comment has been minimized.

RomainFrancony commented Sep 18, 2018

Hi, I came across a problem that may be related to that.
Client can't connect to the control panel, only solution is to clear the stache.

I dig into the problem and concluded to this : users are stored correctly in the stache but the users/data key and other user related keys are not referenced inside the keys file in the stache.
So when you try to login, the server doesn't have any users and therefore the login fails.

Happen on production and some local environment but couldn't find a way to reproduce it.

@jackmcdade

This comment has been minimized.

Member

jackmcdade commented Sep 19, 2018

@RomainFrancony

This comment has been minimized.

RomainFrancony commented Sep 19, 2018

I use 2.10.5 and it happened before in 2.10.4

@efoken

This comment has been minimized.

efoken commented Sep 25, 2018

We're having heavy problems related to this issue, I think ... we use Minio / self-hosted S3 storage for our assets.

After successfull deployment, we clear all our caches (incl. the Stache and Static Cache) and then - just sometimes - all images / content inside the {{ assets:images }} tag disappear ... and sometimes also the navigation disappears - all other content remains.

If this happens, we have to clear all caches multiple times until everything works fine again.

He're some details about the site:

  • ~150 pages
  • approx. 30k assets
  • only 2 collections with ~500 entries
  • very much other content coming from DB
  • ~10k site visits per day
@1stevengrant

This comment has been minimized.

1stevengrant commented Oct 23, 2018

We're seeing this really consistently on a new build that's running the latest Statamic version running PHP 5.6

Will updating PHP version help resolve as we've not seen this on any other Statamic builds lately?

@efoken

This comment has been minimized.

efoken commented Oct 23, 2018

We're currently running PHP 7.2.11 ...

@1stevengrant

This comment has been minimized.

1stevengrant commented Oct 23, 2018

we don't have anywhere near the content levels that you do @efoken

@jackmcdade

This comment has been minimized.

Member

jackmcdade commented Oct 23, 2018

PHP 7+ helps a lot, as does Statamic 2.10.7. Highly recommend being on both, and then letting us know if you have issues.

@1stevengrant

This comment has been minimized.

1stevengrant commented Oct 23, 2018

@jackmcdade yeah still seeing this issue it seems after migrating to a new server.

@jackwakefield

This comment has been minimized.

jackwakefield commented Oct 23, 2018

If you're using CACHE_DRIVER=file there's the potential for a race-condition in Laravel 5.1's cache API fixed with illuminate/filesystem@f9bbd81
illuminate/cache@7fda5a5

Similarly for SESSION_DRIVER=file illuminate/session@1fc8f07

@bacheson

This comment has been minimized.

bacheson commented Nov 13, 2018

We are having this exact issue ... cache is corrupt every few hours and we have to manually clear it

@jasonvarga

This comment has been minimized.

Member

jasonvarga commented Nov 13, 2018

What version are you on?

@bacheson

This comment has been minimized.

bacheson commented Nov 13, 2018

What version are you on?

latest...php 7.0.32

@jasonvarga

This comment has been minimized.

Member

jasonvarga commented Nov 13, 2018

Are you running a load balanced setup?

@bacheson

This comment has been minimized.

bacheson commented Nov 13, 2018

Are you running a load balanced setup?

yes, multiple instances are behind a google cloud loadbalancer

@jasonvarga

This comment has been minimized.

Member

jasonvarga commented Nov 13, 2018

Try using a shared Redis instance for caching, if you aren't already.
https://docs.statamic.com/knowledge-base/redis-cache

@bacheson

This comment has been minimized.

bacheson commented Nov 13, 2018

Try using a shared Redis instance for caching, if you aren't already.
https://docs.statamic.com/knowledge-base/redis-cache

funny you should mention...redis doesn't appear to be working either...i get a redis timeout error even though the correct host/port is specified in the .env file....i can communicate using redis-cli on the same machine

APP_ENV=dev
CACHE_DRIVER=redis
REDIS_HOST=redis.development-blakeacheson.svc.cluster.local
REDIS_PORT=6379

fyi this is not some noob post, I use redis with many other systems with far more complex configs than this

@bacheson

This comment has been minimized.

bacheson commented Nov 14, 2018

Not sure if this helps anyone else but we ended up solving the problem by switching our Kubernetes service load balancing to "local". In this scenario requests are trickled into new containers after the node passes health checks.

We believe at this point that there is a cache race condition bug in statamic/laravel that is exploited by pushing parallel traffic at a virgin container. It would explain why our dev and staging environments are fine but production consistently fails.

@1stevengrant

This comment has been minimized.

1stevengrant commented Nov 14, 2018

we don't have anything like the setup of load balancer and such but we still see this issue when editors are in and making changes. They're getting logged out, can't get back etc.

@bacheson

This comment has been minimized.

bacheson commented Nov 14, 2018

never mind it's now reared it's ugly head in production once again...we've disabled static cacheing and it's still occurring so there is something really wrong going on in the stache i'm assuming

@jackwakefield

This comment has been minimized.

jackwakefield commented Nov 21, 2018

We believe we might be hitting this consistently editing 1 specific page in the CMS containg a large number of suggest fields, after around the 8th request to /cp/addons/suggest/suggestions we're logged out.

@dthvega

This comment has been minimized.

dthvega commented Nov 29, 2018

Has anyone figured a way to reduce it occurring?
It has been happening a lot with our site in recent days where people would edit things and then all the images would disappear and you are unable to log in.

@Christophvh

This comment has been minimized.

Christophvh commented Nov 30, 2018

Try using a shared Redis instance for caching, if you aren't already.
https://docs.statamic.com/knowledge-base/redis-cache

funny you should mention...redis doesn't appear to be working either...i get a redis timeout error even though the correct host/port is specified in the .env file....i can communicate using redis-cli on the same machine

APP_ENV=dev
CACHE_DRIVER=redis
REDIS_HOST=redis.development-blakeacheson.svc.cluster.local
REDIS_PORT=6379

fyi this is not some noob post, I use redis with many other systems with far more complex configs than this

We have exactly the same issue. Suddenly our site just stopped working , when switching from cache_driver=redis to cache_drive=file it worked again, still nog clue why. We have 2 statamic sites running on the same server using the same redis server tho.

@jasonvarga

This comment has been minimized.

Member

jasonvarga commented Nov 30, 2018

This patch to statamic/core/Stache/Manager.php might solve the issue. At least the getting logged out one. Please let me know.

image

and for copy/pasting:

    {
        if ($this->stache->lock()->acquire(true)) {
            $this->doUpdate();
            $this->stache->lock()->release();
        }
    }

    protected function doUpdate()
@jkindness

This comment has been minimized.

jkindness commented Nov 30, 2018

We are also experiencing this issue a lot. At first it was just randomly not showing images and generally was solved by clearing the cache. But now we are also experiencing this login issue where our users cannot login.

@jackmcdade

This comment has been minimized.

Member

jackmcdade commented Nov 30, 2018

@jkindness try the patch in the comment right above yours.

@jkindness

This comment has been minimized.

jkindness commented Nov 30, 2018

@jackmcdade so far it's working. But I'm sure we will have issues again, especially with the broken images.

@jackmcdade

This comment has been minimized.

Member

jackmcdade commented Nov 30, 2018

Don't be so sure ;)

@efoken

This comment has been minimized.

efoken commented Dec 4, 2018

@jackmcdade @jkindness Still having this issues ;-) with 2.11.3

@jackmcdade

This comment has been minimized.

Member

jackmcdade commented Dec 4, 2018

@efoken can you detail your specific issues in a new Issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment