Mails do not get sent to users when exim is used as mail daemon (sendmail with -t command). #4866

Closed
pspoerri opened this Issue Aug 23, 2013 · 27 comments

Projects

None yet
@pspoerri
  1. Summary: Mails do not get sent to users when exim is used as mail daemon.

  2. Steps to reproduce: Setup gitlab (5.2 and 6.0) on linux with exim as sendmail client.

  3. Expected behavior: Mail should arrive at users location.

  4. Observed behavior: Gitlab will notify that a mail to $user has been sent but exim won't report that it sent a mail to $user. Exim will also report

    From: Mail Delivery System <Mailer-Daemon@example.com>
    Subject: Mail failure - no recipient addresses
    Date: 29. Juli 2013 13:51:23 MESZ
    To: <ops@example.com>
    A message that you sent using the -t command line option contained no
    addresses that were not also on the command line, and were therefore
    suppressed. This left no recipient addresses, and so no delivery could
    be attempted.
    
  5. Output of checks

    • Results of GitLab [Application Check]

      # bundle exec rake gitlab:check RAILS_ENV=production
      Checking Environment ...
      Git configured for git user? ... yes
      Has python2? ... yes
      python2 is supported version? ... yes
      Checking Environment ... Finished
      Checking GitLab Shell ...
      GitLab Shell version >= 1.7.0 ? ... OK (1.7.0)
      Repo base directory exists? ... yes
      Repo base directory is a symlink? ... no
      Repo base owned by git:git? ... yes
      Repo base access is drwxrws---? ... yes
      post-receive hook up-to-date? ... yes
      post-receive hooks in repos are links: ...
      Checking GitLab Shell ... Finished
      Checking Sidekiq ...
      Running? ... yes
      Checking Sidekiq ... Finished
      Checking GitLab ...
      Database config exists? ... yes
      Database is SQLite ... no
      All migrations up? ... yes
      GitLab config exists? ... yes
      GitLab config outdated? ... no
      Log directory writable? ... yes
      Tmp directory writable? ... yes
      Init script exists? ... no
      Try fixing it:
      Install the init script
      For more information see:
      doc/install/installation.md in section "Install Init Script"
      Please fix the error above and rerun the checks.
      Init script up-to-date? ... can't check because of previous errors
      Projects have satellites? ...
      [...]
      Redis version >= 2.0.0? ... yes
      Your git bin path is "/usr/bin/git"
      Git version >= 1.7.10 ? ... no
      Try fixing it:
      Update your git to a version >= 1.7.10 from 1.7.9
      Please fix the error above and rerun the checks.
      

      I know that git has not been updated yet. But it doesn't have anything to do with gitlab email. Hence I'm leaving it there.

    1. Possible fixes:
        diff --git a/config/application.rb b/config/application.rb
        index f7349fa..75eee2c 100644
        --- a/config/application.rb
        +++ b/config/application.rb
        @@ -11,6 +11,8 @@ end

         module Gitlab
           class Application < Rails::Application
        +    config.action_mailer.sendmail_settings = { :arguments => "-i" }
        +
             # Settings in config/environments/* take precedence over those specified he
             # Application configuration should go into files in config/initializers
             # -- all .rb files in that directory are automatically loaded.

This issue was also discussed in #3943. I thought to reopen it but since it was closed by the requestor. I thought it might be better to just create a new issue.

The issue is also discussed here: mikel/mail#70

@lfaraone
lfaraone commented Oct 3, 2013

From the exim4 manpage for sendmail:

   -t        When  Exim is receiving a locally-generated, non-SMTP message
             on its standard input, the -t option causes the recipients of
             the message to be obtained from the To:, Cc:, and Bcc: header
             lines in the message instead of from the  command  arguments.
             The  addresses are extracted before any rewriting takes place
             and the Bcc: header line, if present, is then removed.

             If the command has any arguments, they specify  addresses  to
             which  the message is not to be delivered. That is, the argu-
             ment addresses are removed from the recipients list  obtained
             from  the  headers. 

GitLab should either not specify the destination addresses on the command line or not pass -t to sendmail.

@0x20h
0x20h commented Oct 14, 2013

Same issue here. We applied the fix and it works now, thanks.

@JCMais
JCMais commented Oct 15, 2013

Same issue here, the fix did not work for me.
Adding extract_addresses_remove_arguments = false to the exim configuration worked, but I don't know if it can cause any other issues.

@ghost
ghost commented Oct 18, 2013

Fix alone wasn't enough for me (even after GitLab restart)
With the exim option "extract_addresses_remove_arguments = false" and the restart of exim, it worked.

@JCMais
JCMais commented Oct 18, 2013

Just to reiterate my last comment, the fix in reality worked, the problem was that I had not restarted the gitlab service (dumb me).

@dominikschulz

Same problem here. Please fix.

@LeviSchuck

Still a problem.

@keyneom
keyneom commented Jan 18, 2014

Still a problem as of 6-4 stable. I confirmed the fix (using "-i" as the only parameter) fixes the issue.

@mologie
mologie commented Jan 19, 2014

I can also confirm this issue on Debian 7 with exim4.

@lfaraone

Hi all. Yes, this is a known issue. We don't need additional confirmation; its a bug in either upstream Ruby or Exim4, depending on how you see it :)

Please don't comment just to say you've verified it.

@demofly
demofly commented Mar 25, 2014

Same issue, thanx a lot!
Hope, this fix will be added into official FAQ or HOWTO

@lschoepps lschoepps referenced this issue in sbadia/puppet-gitlab Jun 12, 2014
Merged

Fix for compatibility issue with exim #159

@randx randx added a commit that referenced this issue Jul 25, 2014
@randx randx Merge branch 'exim-problems' into 'master'
Exim problems

After merge report back to #4866

See merge request !990
5a89b61
@hanoii
hanoii commented Aug 11, 2014

If there's an actual fix that can be done within gitlab, isn't it worth fixing it there as well while the upstream issue is sorted out?

@dosire
Member
dosire commented Aug 12, 2014

@hanoii its a bug in either upstream Ruby or Exim4, we won't change GitLab to work around it since that can break other clients

@pspoerri

@dosire unfortunately I don't have time to add this to the trouble shooting guide at the moment. So it would be good if someone else could publish it there :)
The issue was fixed in the bug I referenced in the original posting above: mikel/mail#477 I haven't looked at the internals of Gitlab but I think that a similar solution might be feasible in Gitlab. Or using a library for that matter...

These kind of issues are really annoying from a sysops/deployment perspective. Since every little tweak adds additional layers of difficulties during software upgrades. This is especially annoying when the program (i.e. gitlab) is managed by puppet or salt.

@dosire
Member
dosire commented Aug 12, 2014

@minusinf I understand the problem but every little tweak in GitLab also introduces problems, so it is a tradeoff.

@yoe
yoe commented Jul 6, 2015

The exim 4 info documentation has this to say about -t:

 If the command has any arguments, they specify addresses to which
 the message is _not_ to be delivered. That is, the argument
 addresses are removed from the recipients list obtained from the
 headers. This is compatible with Smail 3 and in accordance with
 the documented behaviour of several versions of Sendmail, as
 described in man pages on a number of operating systems (e.g.
 Solaris 8, IRIX 6.5, HP-UX 11). However, some versions of Sendmail
 _add_ argument addresses to those obtained from the headers, and
 the O'Reilly Sendmail book documents it that way. Exim can be made
 to add argument addresses instead of subtracting them by setting
 the option `extract_addresses_remove_arguments' false.

IOW, it's not a bug in exim, it's just a matter of badly specified "standards", and conflicting defaults. The fix is to, simply, add a line extract_addresses_remove_arguments = false to the global section of exim's configuration file...

I've been running gitlab with exim in this configuration for about a year now, with no issues

@pravi
Contributor
pravi commented Jul 23, 2015

Can this workaround be mentioned in the installation document? Switching to postfix for just one line configuration change seems not worth it.

@dosire
Member
dosire commented Jul 23, 2015

@pravi Feel free to suggest a non-intrusive merge request to change the manual installation doc.

@isti37
isti37 commented Jul 25, 2015

extract_addresses_remove_arguments = false in /etc/exim/exim.conf saved the day because I couldn't get smtp either to work with gitlab.

@emmenlau
emmenlau commented Oct 6, 2015

On Debian and Ubuntu, with a split exim4 config, the following worked for me:
Create a new file /etc/exim4/conf.d/main/10_exim4-gitlab_config_options with contents

# GitLab workaround against a problem how exim4 handles the -t option:
extract_addresses_remove_arguments=false

Then regenerate configs and restart exim4 with:

sudo update-exim4.conf
sudo service exim4 restart

If you already have frozen mails that could not be delivered before, you can send the whole queue with

sudo exim4 -qff
@yoe
yoe commented Oct 6, 2015

I'd just like to add that the bug really is in gitlab for using a command-line option that is not necessary and has wildly different results for different MTA's. There are plenty of MUA's which support sending mail with sendmail that this shouldn't be an issue.

The proper fix, really, is to either drop use of the -t command-line option and to specify all recipient addresses to which mail should be sent on the command line, or to retain it but then to specify all recipient addresses in the mail rather than on the command line.

Anything else is not portable and should be avoided.

@chewi
chewi commented Oct 27, 2015

I just ran into this myself so yes, it is annoying, but let's be clear, it is a Rails issue. It was also an issue in mikel/mail#70 but that's since been fixed. Even in current master, Rails overrides the arguments used by Mail here. I think rails/rails#1755 should be reopened.

@jvanbaarsen
Contributor

We are currently in the process of closing down the issue tracker on GitHub, to prevent duplication with the GitLab.com issue tracker. Since this is an older issue I'll be closing this for now. If you think this is still an issue I encourage you to open it on the GitLab.com issue tracker.

@jvanbaarsen jvanbaarsen closed this Jan 7, 2016
@shane-axiom

Still relevant, migrated to GitLab's issue tracker here: https://gitlab.com/gitlab-org/gitlab-ce/issues/12754

@dosire
Member
dosire commented Jan 26, 2016

Thanks for moving @shane-axiom

@chewi
chewi commented Jun 22, 2016

FYI this is fixed in Rails master and the 4.2 branch so it should hit 4.2.7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment