New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance issues that causing users abandon mautic. #3358

Closed
FarhadF opened this Issue Feb 7, 2017 · 26 comments

Comments

Projects
None yet
@FarhadF
Copy link
Contributor

FarhadF commented Feb 7, 2017

What type of report is this:

Q A
Bug report? No
Feature request? No
Enhancement? Yes

Description:

The Problem: Slow processing for large lists. link

Problem Definition:

  1. Getting 100% CPU usage on one core, Due to Single threaded PHP and using a single instance of mautic:campaign:trigger.
    How to solve:
    We need to implement multithreading, for this script, or split the lists of leads and send them to forks.
    I personally prefer the second option because it doesnt require users to recompile php.

  2. Emails/Campaigns need priority
    Imagine the first problem is solved, and mautic is configured to use queue mode. having 3 campaigns to send 250k emails each, Users need Priority for a forth campaign that sending transnational emails. like sign up notifications an stuff.

Priorities: Issue number one is high priority at the moment in my opinion.

My configuration:

  • Self Hosted
  • 4vCPUs, Doesnt matter unfortunately :(
  • 8GB RAM
  • 230K Subscribers
  • Using Queue
  • 8Hours Send time
  • Using Postfix MTA with 128 Public Ips, Benchmarked for being able to send about 800,000 Emails/Hour
@AdrianMackenzieQlsso

This comment has been minimized.

Copy link

AdrianMackenzieQlsso commented Mar 7, 2017

Same Issue with my Mautic

@zenzozen

This comment has been minimized.

Copy link

zenzozen commented Mar 14, 2017

same here. horrible performance with 200k list

@ninjoan

This comment has been minimized.

Copy link

ninjoan commented Apr 4, 2017

Any update?
I dont have the CPU usage problem but i have the speed problem when i try to go from one page to other sample (Dashboard > Emails)

@dbhurley dbhurley added the Backlog label Nov 27, 2017

@dbhurley dbhurley closed this Nov 28, 2017

@howlinghuffy

This comment has been minimized.

Copy link
Contributor

howlinghuffy commented Nov 30, 2017

Same issue here. We are having terrible performance issues with a list of about 250,000. Even opening the emails tab takes about 10 minutes to fully load, and trying to modify a campaign often times out completely. Anyone know where these bottlenecks are coming from?

@SmittyTan

This comment has been minimized.

Copy link

SmittyTan commented Jun 22, 2018

it looks like I am going to abandon Mautic Now after spending 2 days on this!

@npracht

This comment has been minimized.

Copy link
Member

npracht commented Jun 22, 2018

@SmittyTan come and chat on Slack please ;)

@SmittyTan

This comment has been minimized.

Copy link

SmittyTan commented Jun 22, 2018

and your slack chat is DOWN!

@SmittyTan

This comment has been minimized.

Copy link

SmittyTan commented Jun 22, 2018

https://www.mautic.org/slack/ "service unavaialble"

@npracht

This comment has been minimized.

Copy link
Member

npracht commented Jun 22, 2018

then you have my email address on my Github profile, just drop me an email ;)

@SmittyTan

This comment has been minimized.

Copy link

SmittyTan commented Jun 22, 2018

@Gp2mv3

This comment has been minimized.

Copy link

Gp2mv3 commented Jul 23, 2018

Is there a solution for those super slow requests ?
We are using Mautic for some month now and the interface is super super slow (20-30 sec per link click).

I also discovered that the server load is increasing with time (See CPU usage and load average).
screenshot_2018-07-23 mautic datadog
We upgraded our cloud instance on the 10th to a 2 CPUs at 2.3GHz with 7GB of RAM, which explains the sudden change.

We use apache2 with php fpm-7.0. We have op cache and all the setup requirements are met. We have around 100k contacts and 3 campaigns running with around 100 emails sent per day via Sendgrid, not that much I think.

What's wrong with that ? Is it normal that we need super servers to handle that few contacts ?

@escopecz

This comment has been minimized.

Copy link
Member

escopecz commented Jul 23, 2018

@Gp2mv3 can you specify what requests are that slow? Dashboard, email list, tracking, ...?

@zenzozen

This comment has been minimized.

Copy link

zenzozen commented Jul 23, 2018

@Gp2mv3 please let me know how you handle it, if possible. Im working with a very expensive server for 300k contacts too. Same problem with slow interfaces in general. And impossible Reports (i installed mysql workbench and made my own reports directely from database). I´ve already signup a contract with another company, but i want to keep using mautic too for some small things.

@Gp2mv3

This comment has been minimized.

Copy link

Gp2mv3 commented Jul 23, 2018

@escopecz Everything in the menu of the dashboard, every edit, every new page loading,...
Here is an example of timing for /s/segments:
screenshot from 2018-07-23 18-24-21

@zenzozen Currently we don't really handle it, we work slowly... The lags are from Mysql for you ?

@escopecz

This comment has been minimized.

Copy link
Member

escopecz commented Jul 24, 2018

Mautic can load some counts over stats on list pages, it can load some stats on detail pages, tracking requests might wait for some triggers like webhooks that can be slow, but even edit pages? Those doesn't load anything from stats tables which can be huge. There must be some bottleneck in your infrastructure.

Could you profile it with a tool like https://blackfire.io to get more details on what's taking so long?

@Gp2mv3

This comment has been minimized.

Copy link

Gp2mv3 commented Jul 24, 2018

I've cachegrind profiling files. How can I send them to you ?

@escopecz

This comment has been minimized.

Copy link
Member

escopecz commented Jul 24, 2018

Can you publish them here so we can think about solution as a community? There may be others that can spot the problem better than me alone.

@Gp2mv3

This comment has been minimized.

Copy link

Gp2mv3 commented Jul 24, 2018

Yep, I wasn't sure about the sensitivity of those files but it seems they don't contain any sensitive data.
Here they are (In a zip): cachegrind-mautic.zip

@AlbanL74fr

This comment has been minimized.

Copy link

AlbanL74fr commented Jul 24, 2018

When my company started using Mautic, we discovered that we could not install the tracking code on all the pages of our website because the volume of https requests resulting of every page hit would really slow Mautic frontend and make it almost unusable... Slowness correlated with visit peeks... CPU was always 100%. Therefore, we started tracking only pages related to our marketing campaigns. We have about 1M monthly visits on this website.
Another thing that we did to mitigate the risk of slow frontend was to ensure CRON jobs are not running at the same time.

@Gp2mv3

This comment has been minimized.

Copy link

Gp2mv3 commented Jul 24, 2018

@AlbanL74fr Our crons are running every 15m and none are at the same time. And we don't use tracking. Mautic is only used to send emails. Contacts are created via the API called from our servers (around 1k times a day).

@escopecz

This comment has been minimized.

Copy link
Member

escopecz commented Jul 24, 2018

@Gp2mv3 I'm not able to find the bottleneck from the cachegrind reports. I don't see any database calls there. Only that the most costly operation is rendering of PHP templates. Could you try the Blackfire service and link the report of a long loading page here?

@AlbanL74fr take a look at https://github.com/mikegillis677/documentation/blob/5207b12813b897f73d6062b85e28627196789f2c/en/queue/README.md. The PR is missing new menu item so it's not showing up in the Mautic docs...

@Gp2mv3

This comment has been minimized.

Copy link

Gp2mv3 commented Jul 24, 2018

Email edit: https://blackfire.io/profiles/f3fe5d6e-9b60-4597-9cf8-4bf5d25cd4bd/graph
Contacts list: https://blackfire.io/profiles/69d06e2a-d75e-4372-beb6-bf04c3968b7f/graph

I don't understand why the pages loads faster in Blackfire than in my browser...

@escopecz

This comment has been minimized.

Copy link
Member

escopecz commented Jul 25, 2018

It profiles only the backend processing. It will take browser some time to render it and run all the javascript.

So from what I see; SQL queries aren't the blocker. It takes 1/3 of the time for Symfony to send the actual content that the controller prepares to be sent. I tried to google around to find if this is a common problem, but did not find anything.

screen shot 2018-07-25 at 09 42 29

@Gp2mv3

This comment has been minimized.

Copy link

Gp2mv3 commented Jul 25, 2018

@escopecz Yes, I understand that. When I say "I don't understand why the pages loads faster in Blackfire than in my browser..." I refer to the timing graph I sent before, the "Waiting" line is mainly composed of the backend processing, which should be pretty similar to the timings in blackfire.

What I see is that the Waiting is around 2 - 5 seconds and blackfire is around 1s. That seems weird for me...

@Gp2mv3

This comment has been minimized.

Copy link

Gp2mv3 commented Jul 27, 2018

Ok, I think I found the issue. We now have a normal response time.

In Blackfire, it says xDebug is still enabled (which is not the case). I found a thread on stackoverflow referring to a "bug" in php that enables the extension even when it's loaded but not enabled.

So I had to remove the line zend_extension = "/path/to/php_xdebug.so" from my config files.

Here is the thread: https://stackoverflow.com/questions/8754826/how-to-disable-xdebug

@ston3o

This comment has been minimized.

Copy link

ston3o commented Dec 10, 2018

EDIT: Sorry, It's a server configuration problem, I reduced loading time with opcache ;)

Hello @Gp2mv3, I use mautic:2.15.0-beta and I had always performance issue.
Here blackfire report :

blackfire report

Composer\Autoload\includeFile function seems to be the problem.

Mautic takes 3.72s to start

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment