need Mastodon outgoing email diagnostics

[woozalia]

My instance suddenly stopped sending emails about a week ago, and I don't know why. I've tried 2 different SMTP servers; both work when I use them to send email from Thunderbird, both respond to hails on the appropriate port when I connect from Mastodon's server via telnet, but mail doesn't come through from Mastodon regardless.

What I mainly need is some way to get more diagnostic information out of Mastodon.

1. I get that the mastodon-web service logs are accessed with "journalctl -u mastodon-web" -- but this doesn't show where the email was being sent or even whether it was successful.
2. Where is there a log file or queue which shows email subject, destination address, and (especially) whether successful or not, and any associated error messages?
3. Is there a command to interactively send a test email, so you can get status information directly as the attempt is being made?
4. Can Mastodon be configured to send email without needing SMTP relay server credentials (the way MediaWiki and most PHP apps do, using the PHP mail() function)?
5. What file and line of Ruby code is where the email is actually sent? (I don't know Ruby yet, but am willing to learn in order to fix this problem. I'll write my own email logging code if there really aren't any other options.)

Thanks in advance for any help with this.

[nightpool]

the emails are sent from the sidekiq workers, not the web process.
journalctl -u mastodon-sidekiq

while rails does have support for bare sendmail commands, most users do not go this route, due to the complexities of setting up a mail server that will reliably and consistently deliver to major email providers.

[nightpool]

the emails are sent from the sidekiq workers, not the web process.
journalctl -u mastodon-sidekiq

while rails does have support for bare sendmail commands, most users do not go this route, due to the complexities of setting up a mail server that will reliably and consistently deliver to major email providers.

[woozalia]

Thanks. Progress so far:

journalctl -u mastodon-sidekiq | tail -f will get me the last few lines, which is helpful but does not ever seem to show any email-related activity. Probably I'm just not fast enough switching screens from the web browser to the terminal and then requesting another dump, and the relevant events are already pushed out of the window (looks to be 10 lines).

However... 'journalctl -u mastodon-sidekiq | grep smtp' returns a bunch of stuff which may help with tracking down where in the code the emails are sent but still does not have necessary diagnostic information.

So, the following issues remain:

1. (r1) 'journalctl -u mastodon-sidekiq | grep smtp' yields a sequence of events associated with sending emails -- but this doesn't show where the email was being sent or even whether it was successful.
2. Where is there a log file or queue which shows email subject, destination address, and (especially) whether successful or not, and any associated error messages?
3. Is there a command to interactively send a test email, so you can get status information directly as the attempt is being made?
4. (r1) How can Mastodon be configured to send email without needing SMTP relay server credentials (the way MediaWiki and most PHP apps do, using the PHP mail() function)?
5. (r1) What file and line of Ruby code is where the email is actually sent? (The log output suggests some places to look, at least.)

Side note: I've never found any need to do configuring for outgoing email on a properly-connected server; it's incoming that is the bear, while outgoing Just Works (except for Microsoft email servers arbitrarily blocking certain IPs, but that's a social/plutonomy issue, not a technical one).

[woozalia]

Thanks. Progress so far:

journalctl -u mastodon-sidekiq | tail -f will get me the last few lines, which is helpful but does not ever seem to show any email-related activity. Probably I'm just not fast enough switching screens from the web browser to the terminal and then requesting another dump, and the relevant events are already pushed out of the window (looks to be 10 lines).

However... 'journalctl -u mastodon-sidekiq | grep smtp' returns a bunch of stuff which may help with tracking down where in the code the emails are sent but still does not have necessary diagnostic information.

So, the following issues remain:

1. (r1) 'journalctl -u mastodon-sidekiq | grep smtp' yields a sequence of events associated with sending emails -- but this doesn't show where the email was being sent or even whether it was successful.
2. Where is there a log file or queue which shows email subject, destination address, and (especially) whether successful or not, and any associated error messages?
3. Is there a command to interactively send a test email, so you can get status information directly as the attempt is being made?
4. (r1) How can Mastodon be configured to send email without needing SMTP relay server credentials (the way MediaWiki and most PHP apps do, using the PHP mail() function)?
5. (r1) What file and line of Ruby code is where the email is actually sent? (The log output suggests some places to look, at least.)

Side note: I've never found any need to do configuring for outgoing email on a properly-connected server; it's incoming that is the bear, while outgoing Just Works (except for Microsoft email servers arbitrarily blocking certain IPs, but that's a social/plutonomy issue, not a technical one).

[woozalia]

I seem to have fixed the problem. A key fact that was not mentioned is that you have to restart the SideKiq service, possibly in addition to the web service. I was assuming that .env.production was read by the web service and SideKiq just read the settings from there -- but whether or not this is correct, SideKiq grabs them once when it starts up and ignores any changes after that.

A revised issue list:

1. (r1) journalctl -u mastodon-sidekiq | grep smtp yields a sequence of events associated with sending emails -- but this doesn't show where the email was being sent or even whether it was successful.
2. Where is there a log file or queue which shows email subject, destination address, and (especially) whether successful or not, and any associated error messages? (Tentatively: there isn't one.)
3. (r1) It looks like there is no script solely for sending test emails, but the /lib/tasks/mastodon.rake task does include code for doing this; it's just that you can only run it as part of a longer setup script. That piece should be extracted such that a test email could be sent based on the current .env.production settings instead of requiring them to be entered (along with lots of other stuff) each time.
4. (SOLVED) Mastodon can be configured to send email without needing SMTP relay server credentials using settings found in that same .rake file, to wit:

> SMTP_SERVER=localhost
> SMTP_PORT=25
> SMTP_AUTH_METHOD=none
> SMTP_OPENSSL_VERIFY_MODE=none

5. (r2) What file and line of Ruby code is where the email is actually sent? PARTIAL SOLUTION: Ruby library SMTP, located (on our server) in /home/mastodon/.rbenv/versions/2.5.0/lib/ruby/2.5.0/net/smtp.rb, module Net (aka Net::SMTP), class SMTP, function do_start() seems to do the actual network operations. It's not clear where that is being called from, but some basic information about each attempt could presumably be found there.

Until I come across evidence that there are still problems with email on my instance, I'm de-prioritizing my continued work on this issue -- but there are still questions to be answered and documented.

[woozalia]

I seem to have fixed the problem. A key fact that was not mentioned is that you have to restart the SideKiq service, possibly in addition to the web service. I was assuming that .env.production was read by the web service and SideKiq just read the settings from there -- but whether or not this is correct, SideKiq grabs them once when it starts up and ignores any changes after that.

A revised issue list:

1. (r1) journalctl -u mastodon-sidekiq | grep smtp yields a sequence of events associated with sending emails -- but this doesn't show where the email was being sent or even whether it was successful.
2. Where is there a log file or queue which shows email subject, destination address, and (especially) whether successful or not, and any associated error messages? (Tentatively: there isn't one.)
3. (r1) It looks like there is no script solely for sending test emails, but the /lib/tasks/mastodon.rake task does include code for doing this; it's just that you can only run it as part of a longer setup script. That piece should be extracted such that a test email could be sent based on the current .env.production settings instead of requiring them to be entered (along with lots of other stuff) each time.
4. (SOLVED) Mastodon can be configured to send email without needing SMTP relay server credentials using settings found in that same .rake file, to wit:

> SMTP_SERVER=localhost
> SMTP_PORT=25
> SMTP_AUTH_METHOD=none
> SMTP_OPENSSL_VERIFY_MODE=none

5. (r2) What file and line of Ruby code is where the email is actually sent? PARTIAL SOLUTION: Ruby library SMTP, located (on our server) in /home/mastodon/.rbenv/versions/2.5.0/lib/ruby/2.5.0/net/smtp.rb, module Net (aka Net::SMTP), class SMTP, function do_start() seems to do the actual network operations. It's not clear where that is being called from, but some basic information about each attempt could presumably be found there.

Until I come across evidence that there are still problems with email on my instance, I'm de-prioritizing my continued work on this issue -- but there are still questions to be answered and documented.

[nightpool]

what does r1/r2 mean?

[nightpool]

what does r1/r2 mean?

[nightpool]

re: 1, 2: navigating to /sidekiq gives you a list of all failed or currently-failing sidekiq tasks with error messages and full data, including email tasks

re 3: requesting a password reset is easy and cheap, but we probably wouldn't say no to this enhancement

re 4: you can also use "sendmail" type, as mentioned in .env.production.sample under SMTP_DELIVERY_METHOD

re 5: it's kind of complicated, because rails handles all that stuff transparently. UserMailer is the class associated with authentication emails though, so that's maybe what you're looking for?

[nightpool]

re: 1, 2: navigating to /sidekiq gives you a list of all failed or currently-failing sidekiq tasks with error messages and full data, including email tasks

re 3: requesting a password reset is easy and cheap, but we probably wouldn't say no to this enhancement

re 4: you can also use "sendmail" type, as mentioned in .env.production.sample under SMTP_DELIVERY_METHOD

re 5: it's kind of complicated, because rails handles all that stuff transparently. UserMailer is the class associated with authentication emails though, so that's maybe what you're looking for?

[woozalia]

The "r"s are revisions -- to note that the item has been modified in some way.

re 1,2: I spent hours looking at Sidekiq and trying to find any email jobs, but was unable to do so. Any of the following might have made it possible:

• being able to search by content or worker-type
• being able to see the content parsed out into a table or some other more readable form
• a plaintext rendering, which could then have been searched with browser text-search or grepped

I'm thinking some viewers for the Sidekiq API would be a good thing; I'll add that to my "when opportunity arises" queue (i.e. either another issue makes this more urgent again, or other higher-priority tasks get cleared away).

re 3: definitely noted; this also goes in "when opportunity arises".

re 4: Aha! So that's where I should have been looking for options documentation. Didn't occur to me. ...although it should be noted that there are still some things it doesn't explain. like the format of DATABASE_URL.

re 5: UserMailer looks promising, yes.

Thanks.

[woozalia]

The "r"s are revisions -- to note that the item has been modified in some way.

re 1,2: I spent hours looking at Sidekiq and trying to find any email jobs, but was unable to do so. Any of the following might have made it possible:

• being able to search by content or worker-type
• being able to see the content parsed out into a table or some other more readable form
• a plaintext rendering, which could then have been searched with browser text-search or grepped

I'm thinking some viewers for the Sidekiq API would be a good thing; I'll add that to my "when opportunity arises" queue (i.e. either another issue makes this more urgent again, or other higher-priority tasks get cleared away).

re 3: definitely noted; this also goes in "when opportunity arises".

re 4: Aha! So that's where I should have been looking for options documentation. Didn't occur to me. ...although it should be noted that there are still some things it doesn't explain. like the format of DATABASE_URL.

re 5: UserMailer looks promising, yes.

Thanks.

[nightpool]

@woozalia the ability to search or filter the Sidekiq UI is definitely something i've felt the lack of as well, but it's not part of our project. Maybe see if there are any open issues about it on https://github.com/mperham/sidekiq?

[nightpool]

@woozalia the ability to search or filter the Sidekiq UI is definitely something i've felt the lack of as well, but it's not part of our project. Maybe see if there are any open issues about it on https://github.com/mperham/sidekiq?