Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'Hung' sendout #324

Closed
bossanova808 opened this issue Sep 8, 2022 · 26 comments
Closed

'Hung' sendout #324

bossanova808 opened this issue Sep 8, 2022 · 26 comments
Labels
bug Something isn't working

Comments

@bossanova808
Copy link

Hi Ben

(Craft 3, latest campaign).

Nothing major, but I have just noticed a 'Hung' sendout. It's stuck in our list, with a pending status (unlike all our others):

image

It seems to think there's one more recipient or something:

image

There's a matching report, which seems fine,and shows that it was sent to the expected number:

image

...bit mystified. But it seems to have sent as expected, and all I really would like is to clean this up somehow (...but don't want to lose the record of it).

Any tips as to what to do here (or guesses as to what happened?). We've sent 130 other sendouts without this issue, so things mostly work extremely well (and I love having the newsletter process integrated with all the rest of our content creation - it's genius).

I dug up the logs and see:

2022-08-06 10:26:46 [][1][info] Sendout "2022_06_08 - Image Science August 2022 Newsletter" initiated by "blah@blah".
...but I don't see the corresponding completed message (but also no other errors)

@bossanova808 bossanova808 added the bug Something isn't working label Sep 8, 2022
@bencroker
Copy link
Collaborator

Can you try pausing the sendout, then resume it to ensure it completes?

@bossanova808
Copy link
Author

bossanova808 commented Sep 8, 2022

Just tried that - no change (although it does say 9452 recipients now, oddly...)

So I clicked on the 'recipients' link (never noticed that before) and am a bit concerned to see it appears to be sending it, slowly but surely, to new people as they subscriber...(this was originally sent a month ago...)..and has been doing that all this month (about 9300 people got it August 6th, then the extra 140 odd, as a trickle over this month).

image

There are no pending jobs in the queue or anything.

This is really quite odd....so, I just rebooted the server. And now it's no longer pending, looks the same as the rest, status sent.

Makes no sense to me at all. I use Andrew's approach for queue runnig (systemctl restart imagescience-queue@.service@{1..2} etc) - but there were definitely no jobs in the queue under the Queue Manager. And can a job just potter along for a month...why wouldn't it just run to the end of the current list and stop?

I mean, everything seems fine now - I only noticed this because we're sending another newsletter in about 12 hours....so will be interesting to see if that one completes!

Gremlins in the system....

@bossanova808
Copy link
Author

(Late here, and off to bed - but shall update tomorrow night after the September newsletter had been sent...)

@bossanova808
Copy link
Author

Right, so the problem has repeated with today's campaign

image

I can see, at Postmark, that sending has stopped...all the events now are 'Opened', 'Clicked' etc - no more 'Delivered' - but the Sendout remains in pending status as you can see, and I find those figures just weird - what is that 'Expected Recipients - 1' about?

In the campaign.log there is a start message, a few bounces (as normal although I note each appears twice) - but no complete message:

2022-09-08 15:58:51 [][1][info] Sendout "2022_09_09 - September 2022 Newsletter" initiated
2022-09-09 09:15:03 [][][info] Contact xxx marked as "bounced".
2022-09-09 09:15:05 [][][info] Contact xxx marked as "bounced".
2022-09-09 09:44:43 [][][info] Contact yyy marked as "bounced".
2022-09-09 09:44:45 [][][info] Contact yyy marked as "bounced".
etc... 

The last 'Sent' is about 15 minutes ago, but I bet that what will happen from here is that the Sendout will get sent to the next new user who joins (as happened all last month!). And on from there, if it doesn't think it is finished. So...clearly want to avoid that. Pausing and resuming again did not help (it says 'schedule to send to 1 contact' with that - I presume that's the 'expected recipient' as above?? And again, all jobs are gone from the queue.

Obviously I can reboot the server again at some point, but that's hardly a long term solution. However, am just about to go on holiday for 10 days, so that might be the only choice for now really.

Happy to help you debug after I am back, if you can't think of anything. (Thinking about it, I presume it's the the cron thing that is keeping this running the entire month, combined with the failure of the status to get to 'completed'...)

(BTW you might want to add some sort of lazy load to that 'Recipients' view - it takes an age to load, when there are a lot of recipients - we're talking over a minute...)

@bencroker
Copy link
Collaborator

What specific version of the plugin are you running?

@bossanova808
Copy link
Author

bossanova808 commented Sep 9, 2022 via email

@bencroker
Copy link
Collaborator

My guess would be that that last recipient is making the sendout fail, thereby preventing it from completing. What are the last log messages in campaign.log?

(BTW you might want to add some sort of lazy load to that 'Recipients' view - it takes an age to load, when there are a lot of recipients - we're talking over a minute...)

Do you mean the "expected recipients"? That should load in the background via AJAX, so as to not stall loading of the page.

@bossanova808
Copy link
Author

Nothing but the above - the initiated message, and a few bounces.

And now I see the August one is back to pending and just sent to our latest new user - ergh! It's like a zombie newsletter!

Is there anything I can do, other than deleting the sendouts, to stop them? If I disable the cron, then other Campaigns won't send, right?

(I mean the recipients view that comes from clicking this:

image

@bossanova808
Copy link
Author

I see 'cancel' is an option in the element index...I tried that, it still shows 'pending' (orange triangle) in the index, but 'cancelled' in the sendout itself (red circle).

@bencroker
Copy link
Collaborator

I see 'cancel' is an option in the element index...I tried that, it still shows 'pending' (orange triangle) in the index, but 'cancelled' in the sendout itself (red circle).

It sounds like you're coming up against an issue with Craft's element index caching. Are you on the latest Craft version 3?

@bossanova808
Copy link
Author

bossanova808 commented Sep 9, 2022 via email

@bencroker
Copy link
Collaborator

If you're seeing that the sendout is cancelled on the sendout edit page, then that is the true value. You can verify this in the campaign_sendouts database table. Try clearing Craft's caches, perhaps that will cause the Sendouts index page to refresh.

@bossanova808
Copy link
Author

bossanova808 commented Sep 9, 2022

Yes, it was showing cancelled in the table (the older of the two). I got a bit ahead of you, and decided to change it to sent as per all the others. It's now showing as the regular green circle and sent as the status in the sendout. I didn't clear the cache, it just updated to that at some point (presume when the cron next ran?).

The other one I left for now, just to observe what happens - but it's still at 'pending' (index and in the sendout). So I am inclined to do the same with it. That will tidy these two up, but of course doesn't stop the issue coming around again on the next sendout...

@bencroker
Copy link
Collaborator

Yes, well let me know if this comes up again and we can try to debug the issue on your end.

@bossanova808
Copy link
Author

Will do. I have changed the other one to 'sent' in the table as well.

Thanks Ben.

@bossanova808
Copy link
Author

Hey Ben

So we did another send out today - and the same thing has happened. It says 100%, but I don't get the complete notification and the campaign still has the pending orange triangle...

In the logs I have the starting message, a few notifications of bounces, and that's it - no finished message.

image

And the status in the DB is pending, as per the others above:

image

I can again change that manually to 'sent' but of course would like to get to the bottom of this more properly...

@bencroker
Copy link
Collaborator

Are you able to update to version 2, which requires Craft 4, or are you stuck on Craft 3?

@bencroker bencroker reopened this Oct 11, 2022
@bossanova808
Copy link
Author

Stuck on 3 for a while yet I'd say. Migrating a large, complex, commerce a site is no easy task! And it's the busiest time of year for the actual business, plus staffing in current times is... Hard. So whilst I very much want to get to 4 it depends on how much time I can find along the way...

Late here so off to bed but am willing to do whatever I can to help work it out. But also accept that I can manually mark them as sent until I do get to v4 if it's too hard.

@bencroker
Copy link
Collaborator

bencroker commented Oct 11, 2022

Let's start with figuring our which contact is still pending. Can you add this code to a custom module (change the value of $sendoutId accordingly), which should give you the details of the last remaining contact?

<?php

namespace modules\sitemodule;

use Craft;
use craft\services\Plugins;
use putyourlightson\campaign\Campaign;
use yii\base\Event;
use yii\base\Module;

class SiteModule extends Module
{
    public function init()
    {
        Event::on(
            Plugins::class,
            Plugins::EVENT_AFTER_LOAD_PLUGINS,
            function() {
                $sendoutId = 1;
                Craft::dd(
                    Campaign::$plugin->sendouts->getPendingRecipients(
                        Campaign::$plugin->sendouts->getSendoutById($sendoutId)
                    )
                );
            }
        );
    }
}

@bossanova808
Copy link
Author

Ok so I popped this in a console command and just ran it. I have found the (just one) contact in question.

They have a status of 'Blocked' but I can't see why...the activity log seems normal, and I can't find anything at Postmark either - although their logs are not infinite I think.

image

image

17th of July lines up with roughly when I think the problems actually started...

So - the questions are:

  • what's best to do with this contact - delete them maybe?
  • why did this happen?
  • how can we avoid it happening again?

It might we can just delete this one contact and the problem never comes back...happy to put it down to Gremlins for now, and come back to it again if, after deleting this one, it does pop up again?

(I don't want you to spend a bunch of unnecessary time on one weirdness, really!)

@bossanova808
Copy link
Author

Actually have dug up an old email - they asked to be manually unsubscribed. Looks like whoever handled that manually marked them as Blocked rather than just unsubbing them from the list.

Still begs the question as to why that is causing hangs in sendouts.

@bossanova808
Copy link
Author

Whilst in there I unsubscribed them from the list and am now getting null back from that debug above...

@bossanova808
Copy link
Author

...and that has resulted in me getting the complete message, so it was definitely just that one blocked contact causing the hang.

@bencroker
Copy link
Collaborator

Glad to hear it! If the contact's status was blocked then they should not have been counted as a pending contact, so I'll look into releasing a fix for this.

@bencroker
Copy link
Collaborator

bencroker commented Oct 12, 2022

Fixed in b60f24d and released in versions 2.1.9 and 1.23.5. Thanks for your help in tracking this down!

@bossanova808
Copy link
Author

As ever Ben, awesome support, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants