net/smtp: auth problem with yandex.ru

[gopherbot]

by hashcode.ru:

What steps will reproduce the problem?
1. register email account on yandex.ru
2. try to send email with smtp.SendMail, smtp.yandex.ru, the port is 25, from
someName@yandex.ru to otherName@yandex.ru

What is the expected output?
The email was sent without errors


What do you see instead?

503 5.5.4 Error: send AUTH command first.

Which compiler are you using (5g, 6g, 8g, gccgo)?

6g

Which operating system are you using?
Ubuntu 10.4 LTS

Which revision are you using?  (hg identify)
branch: release-branch.r60 (document release.r60.3)

Please provide any additional information below.

The smtp.SendEmail function doesn't works with all russian mail providers. It works only
if you send email from gmail account to gmail account.

[rsc]

Comment 1:

You didn't post a program, but I suspect you are passing
nil as the Auth argument in SendMail.  From the server
response it sounds like you have to authenticate one way
or another.  Try passing the result of smtp.PlainAuth or
smtp.CRAMMD5Auth instead.
Russ

Status changed to WaitingForReply.

[gopherbot]

Comment 2 by hashcode.ru:

Hi! My function
func sendMessage(message Message){
    auth := smtp.PlainAuth(
        "",
        "webmaster@hashcode.ru",
        "password",
        "smtp.yandex.ru",
    )
    err := smtp.SendMail(
        "smtp.yandex.ru:25",
        auth,
        "webmaster@hashcode.ru",
        []string{"hashcode.ru@gmail.com", "nicolas_dev@mail.ru"},
        []byte("This is the email test body."),
    )
    HandleSendMessageError(err)
}
I've tested it on different params. If I use gmail sender and gmail receiver the
function works, if I use gmail sender and other receiver, some receivers takes a message
with error and some with blank message.
Error:
Received: by mail-bk0-f41.google.com with SMTP id y12so2390138bkt.0
        for <nicolas_dev@mail.ru>; Thu, 16 Feb 2012 06:09:29 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=message-id:date:from;
        bh=ZxZxMJM7Iwc/OmykReD4Td3zlhMrWYK4NuP5Od4JY/o=;
        b=hJoqPEuDSKqKDiKxarivufevmQ+SJsmpa5m2gy9TH2kQRmzWYAtXEN7/nwKnr50OuR
         EbY+OEveCpRuGmneX+yBhXBiVvvtjYgVntdLBb/98jiUdqzXn7LGv3QPBNqYh1LJsuR9
         kkHl9gXDjrNPBQ+AA6a2WNPFAAZrOWNkW8pcw=
Received: by 10.205.129.130 with SMTP id hi2mr1212493bkc.98.1329401369686;
        Thu, 16 Feb 2012 06:09:29 -0800 (PST)
Return-Path: <hashcode.ru@gmail.com>
Received: from localhost ([178.162.75.102])
        by mx.google.com with ESMTPS id o7sm13085208bkw.16.2012.02.16.06.09.28
        (version=TLSv1/SSLv3 cipher=OTHER);
        Thu, 16 Feb 2012 06:09:28 -0800 (PST)
Message-ID: <4f3d0e18.879acc0a.3868.1f8d@mx.google.com>
Date: Thu, 16 Feb 2012 06:09:28 -0800 (PST)
From: hashcode.ru@gmail.com
This is the email test body.
If I use as the sender non gmail account, I catch an error in the sendMessage function: 
503 5.5.4 Error: send AUTH command first.

[rsc]

Comment 3:

I don't know what's going on.  Please try making these changes to
src/pkg/net/smtp/smtp.go:
diff -r b984f2b4aa91 src/pkg/net/smtp/smtp.go
--- a/src/pkg/net/smtp/smtp.go  Thu Feb 16 22:04:13 2012 -0500
+++ b/src/pkg/net/smtp/smtp.go  Fri Feb 17 09:26:15 2012 -0500
@@ -11,6 +11,7 @@
 package smtp
 import (
+"fmt"
    "crypto/tls"
    "encoding/base64"
    "io"
@@ -66,6 +67,7 @@
 // cmd is a convenience function that sends a command and returns the response
 func (c *Client) cmd(expectCode int, format string, args
...interface{}) (int, string, error) {
+fmt.Printf("> %s\n", fmt.Sprintf(format, args...))
    id, err := c.Text.Cmd(format, args...)
    if err != nil {
        return 0, "", err
@@ -73,6 +75,10 @@
    c.Text.StartResponse(id)
    defer c.Text.EndResponse(id)
    code, msg, err := c.Text.ReadResponse(expectCode)
+fmt.Printf("< %d %s\n", code, msg)
+if err != nil {
+   fmt.Printf("ERR: %v\n", err)
+}
    return code, msg, err
 }
When I run this program with those changes:
package main
import (
    "fmt"
    "net/smtp"
)
func main() {
    auth := smtp.PlainAuth(
        "",
        "fake@account",
        "password",
        "smtp.gmail.com",
    )
    err := smtp.SendMail("smtp.gmail.com:587",
        auth,
        "rsc@rsc.swtch.com",
        []string{"rsc@rsc.swtch.com"},
        []byte("Here is a message."),
    )
    fmt.Println(err)
}
I get this output:
$ go run x.go
< 250 mx.google.com at your service, [24.61.14.197]
SIZE 35882577
8BITMIME
STARTTLS
ENHANCEDSTATUSCODES
< 220 2.0.0 Ready to start TLS
< 250 mx.google.com at your service, [24.61.14.197]
SIZE 35882577
8BITMIME
AUTH LOGIN PLAIN XOAUTH
ENHANCEDSTATUSCODES
< 535 5.7.1 Username and Password not accepted. Learn more at
5.7.1 http://support.google.com/mail/bin/answer.py?answer=14257
fj4sm6777173vdb.22
< 530 5.5.1 Authentication Required. Learn more at
ERR: 530 5.5.1 Authentication Required. Learn more at
530 5.5.1 Authentication Required. Learn more at
$
Maybe the trace you get will be enough to identify
why AUTH is not being sent or being rejected.
Russ

[dsymonds]

Comment 4:

Labels changed: added priority-later, removed priority-triage.

[rsc]

Comment 5:

The reported case appears not to be authenticating even
though an auth is being passed, and then server replies with
send AUTH command first.  I want to understand the reason
for Go 1, even if we don't fix it for Go 1.

Labels changed: added priority-go1, removed priority-later.

[gopherbot]

Comment 7 by hashcode.ru:

I've added changes to the sources.
The result:
> HELO localhost
< 250 smtp1.mail.yandex.net
> MAIL FROM:<webmaster@hashcode.ru>
< 503 5.5.4 Error: send AUTH command first.
ERR: 503 5.5.4 Error: send AUTH command first.

[gopherbot]

Comment 8 by hashcode.ru:

It was for the yandex account. For the gmail account:
> EHLO localhost
< 250 mx.google.com at your service, [178.162.75.102]
SIZE 35882577
8BITMIME
STARTTLS
ENHANCEDSTATUSCODES
> STARTTLS
< 220 2.0.0 Ready to start TLS
> EHLO localhost
< 250 mx.google.com at your service, [178.162.75.102]
SIZE 35882577
8BITMIME
AUTH LOGIN PLAIN XOAUTH
ENHANCEDSTATUSCODES
> AUTH PLAIN AGhhc2hjb2RlLnJ1QGdtYWlsLmNvbQBIbnA0cmlhc3Np
< 235 2.7.0 Accepted
> MAIL FROM:<hashcode.ru@gmail.com> BODY=8BITMIME
< 250 2.1.0 OK fw2sm9156422bkc.13
> RCPT TO:<hashcode.ru@gmail.com>
< 250 2.1.5 OK fw2sm9156422bkc.13
> RCPT TO:<nicolas_dev@mail.ru>
< 250 2.1.5 OK fw2sm9156422bkc.13
> DATA
< 354  Go ahead fw2sm9156422bkc.13
> QUIT
< 221 2.0.0 closing connection fw2sm9156422bkc.13

[rsc]

Comment 9:

[Repost of comment #8]
It was for the yandex account. For the gmail account:
> EHLO localhost
< 250 mx.google.com at your service, [178.162.75.102]
SIZE 35882577
8BITMIME
STARTTLS
ENHANCEDSTATUSCODES
> STARTTLS
< 220 2.0.0 Ready to start TLS
> EHLO localhost
< 250 mx.google.com at your service, [178.162.75.102]
SIZE 35882577
8BITMIME
AUTH LOGIN PLAIN XOAUTH
ENHANCEDSTATUSCODES
> AUTH PLAIN ...
< 235 2.7.0 Accepted
> MAIL FROM:<hashcode.ru@gmail.com> BODY=8BITMIME
< 250 2.1.0 OK fw2sm9156422bkc.13
> RCPT TO:<hashcode.ru@gmail.com>
< 250 2.1.5 OK fw2sm9156422bkc.13
> RCPT TO:<nicolas_dev@mail.ru>
< 250 2.1.5 OK fw2sm9156422bkc.13
> DATA
< 354  Go ahead fw2sm9156422bkc.13
> QUIT
< 221 2.0.0 closing connection fw2sm9156422bkc.13

[rsc]

Comment 10:

Thanks for these traces.  It looks like yandex.ru is not advertising the fact that it
supports ESMTP, so the mail client does not use EHLO, so it does not learn about all the
extensions that are available, including AUTH.  The server is supposed to say "ESMTP" in
the 220 banner that it opens the connection with, but mx.yandex.ru does not:
$ dial 'tcp!smtp.yandex.com!25'
220 smtp18.mail.yandex.net (Want to use Yandex.Mail for your domain? Visit
http://pdd.yandex.ru)
^D
$
$ dial 'tcp!smtp.gmail.com!25'
220 mx.google.com ESMTP o8sm58736941qan.11
^D
$ 
Notice the "ESMTP" in gmail.com's banner, as compared to yandex.com's.
ESMTP was introduced in 1995, and perhaps there are no servers left that don't support
it.  Maybe we should change the net/smtp package to always say EHLO.  If you say EHLO to
yandex, it does work and list AUTH as one of the supported extensions:
$ dial 'tcp!smtp.yandex.com!25'
220 smtp18.mail.yandex.net (Want to use Yandex.Mail for your domain? Visit
http://pdd.yandex.ru)
EHLO mit.edu
250-smtp18.mail.yandex.net
250-8BITMIME
250-PIPELINING
250-SIZE 42991616
250-STARTTLS
250-AUTH LOGIN PLAIN
250 ENHANCEDSTATUSCODES
^D
$ 
I will try to find out whether it is a bad idea to say EHLO always.  Or maybe if auth !=
nil SendMail should say EHLO always.  You might want to try to contact yandex's
technical support and try to get them to fix their SMTP server greeting.

Status changed to Accepted.

[rsc]

Comment 11:

CL started.

[rsc]

Comment 12:

This issue was closed by revision 2110fad.

Status changed to Fixed.

[gopherbot]

Comment 13 by hashcode.ru:

I've done the changes in the smtp.go.
I am afraid that sending emails is not working correctly yet. I've created test account
on yandex.ru, a username is go-sendmail@yandex.ru, a password is golang. You can use it
for tests
The code is
   auth := smtp.PlainAuth(
        "",
        "go-sendmail@yandex.ru",
        "golang",
        "smtp.yandex.ru",
    )
    err := smtp.SendMail(
        "smtp.yandex.ru:25",
        auth,
        "go-sendmail@yandex.ru",
        []string{"hashcode.ru@gmail.com", "nicolas_dev@mail.ru", "webmaster@hashcode.ru"},
        []byte("This is the email body. Это тестовое сообщение"),
    )
After running the code, I get a blank email on all accounts (on gmail it is not at all)
with 
From: MAILER-DAEMON whom: undisclosed-recipients. There are no errors in the course of
the program.
Also, I noticed that I have a package path "src/pkg/smtp", and you "src/pkg/net/smtp".
Maybe you have a different branch?

[rsc]

Comment 14:

If you have "smtp" and not "net/smtp", you are probably still
using r60.  Please check out the latest Go tree by using
hg clone -r weekly https://code.google.com/p/go go-weekly
and see if that helps.

[rsc]

Comment 15:

I believe the failure you are seeing now is because your message is
not formatted as a mail message: it has no headers.
Try sending something like this instead:
var msg = `From: go-sendmail@yandex.ru
To: hashcode.ru@gmail.com, nicolas_dev@mail.ru
Subject: test message
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="UTF-8"
This is the email body. Это тестовое сообщение
`

[gopherbot]

Comment 16 by hashcode.ru:

Thank you for the clarification, now all works great!