Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GSoC: finagle-smtp - initial #287

Closed
wants to merge 50 commits into from
Closed
Changes from 48 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
27a48e6
A quick&dirty example of SMTP client
suncelesta Mar 12, 2014
7de8295
simple wrappers for javamail and a.c.e.
suncelesta May 6, 2014
d8f4877
New client API
suncelesta May 7, 2014
3684bf5
SMTP commands
suncelesta May 16, 2014
18f43f2
Reply classes
suncelesta May 26, 2014
728d7f9
Dispatcher and transport
suncelesta Jun 3, 2014
5b9e617
Finished codec
suncelesta Jun 9, 2014
507a775
Filters concerning email payload
suncelesta Jun 10, 2014
4410eeb
Receiving server greeting before the session
suncelesta Jun 11, 2014
f5e7604
Reply code refactoring
suncelesta Jun 11, 2014
bed4e94
Minor bugs fix
suncelesta Jun 11, 2014
699d78d
More concise result in simple client
suncelesta Jun 11, 2014
067917c
Full multiline support
suncelesta Jun 11, 2014
14adab8
Refined structure, tests for filters
suncelesta Jun 17, 2014
69aec96
More tests
suncelesta Jun 21, 2014
3664948
Multiline correction and usage doc
suncelesta Jun 24, 2014
8614e7a
Update README.md
selvin Jun 24, 2014
017d516
Update README.md
selvin Jun 24, 2014
074d72d
Update README.md
selvin Jun 24, 2014
a187f29
Update README.md
selvin Jun 24, 2014
571e85a
Update README.md
selvin Jun 24, 2014
41f35c8
Update README.md
selvin Jun 24, 2014
8626554
Merge pull request #1 from selvin/master
suncelesta Jun 25, 2014
f6c8be9
Sending quit command upon service closing
suncelesta Jun 25, 2014
358655c
Fixed issue with build error
suncelesta Jun 25, 2014
adfc22d
EHLO sends domain/address literal
suncelesta Jul 3, 2014
cd9efe4
Copies and EmailBuilder
suncelesta Jul 5, 2014
3ea84a4
Logs
suncelesta Jul 5, 2014
53c9b63
Tests for MailAddress and EmailBuilder
suncelesta Jul 5, 2014
0ea3914
SmtpSimple sends EHLO once in the beginning
suncelesta Jul 7, 2014
597a8c8
..
suncelesta Jul 7, 2014
bc1ac67
resolve conflict
suncelesta Jul 7, 2014
bd2e836
delete unnecessary file
suncelesta Jul 7, 2014
e560f60
Fix accidental mysql test rearrangement
suncelesta Jul 9, 2014
275ec98
Moved Example.scala to finagle-example
suncelesta Jul 9, 2014
7fa4157
Removed left diff on mysql RequestTest
suncelesta Jul 10, 2014
5681625
try to remove end of line diff
suncelesta Jul 10, 2014
ed5ec40
Minor fixes
suncelesta Jul 10, 2014
bd15c38
Merge branch 'finagle-smtp' of https://github.com/suncelesta/finagle …
suncelesta Jul 10, 2014
258be78
Fixed Sender field
suncelesta Jul 11, 2014
178ae73
Fixed README.MD
suncelesta Jul 11, 2014
f2f6a7e
Added scaladoc comments
suncelesta Jul 17, 2014
9b3c003
Added link to RFC
suncelesta Jul 23, 2014
c04fca9
Corrected link to Example in README
suncelesta Jul 28, 2014
a1f96a8
Email headers
suncelesta Aug 1, 2014
78fe18d
Fixed style and logic
suncelesta Aug 8, 2014
a342264
Field names converted to lowercase
suncelesta Aug 11, 2014
caa31e3
DefaultEmail instead of EmailBuilder
suncelesta Aug 11, 2014
8adbf4d
Quit moved to dispatcher
suncelesta Aug 14, 2014
03d9be6
Fixes for all tests to pass
suncelesta Aug 17, 2014
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -0,0 +1,40 @@
package com.twitter.finagle.example.smtp

import com.twitter.logging.Logger
import com.twitter.finagle.smtp._
import com.twitter.finagle.SmtpSimple
import com.twitter.util.{Await, Future}

/**
* Simple SMTP client with an example of error handling.
*/
object Example {
private val log = Logger.get(getClass)

def main(args: Array[String]): Unit = {
// Raw text email
val email = DefaultEmail()
.from_("from@from.com")
.to_("first@to.com", "second@to.com")
.subject_("test")
.text("first line", "second line") //body is a sequence of lines

// Connect to a local SMTP server
val send = SmtpSimple.newService("localhost:2525")

// Send email
val res: Future[Unit] = send(email) onFailure {
// An error group
case ex: reply.SyntaxErrorReply => log.error("Syntax error: %s", ex.info)

// A concrete reply
case reply.ProcessingError(info) => log.error("Error processing request: %s", info)
}

This comment has been minimized.

Copy link
@mosesn

mosesn Aug 6, 2014

Contributor

this should match the indentation level of the line that started the block.


log.info("Sending email...")

Await.ready(res)

This comment has been minimized.

Copy link
@selvin

selvin Jul 29, 2014

isn't it necessary to block? otherwise this thread ends and since all other threads are daemon threads, the program exits shortly thereafter

This comment has been minimized.

Copy link
@suncelesta

suncelesta Jul 31, 2014

Author

That was what I meant, I just supposed that in practice the program would probably not end like this, but have something else done in background.


log.info("Sent")
}
}
@@ -0,0 +1,90 @@
# finagle-smtp

This is a minimum implementation of SMTP client for finagle according to
[`RFC5321`][rfc]. The simplest guide to SMTP can be found, for example, [here][smtp2go].

Note: There is no API yet in this implementation for creating
[`MIME`][mimewiki] messages, so the message should be plain US-ASCII text, or converted
to such. There is currently no support for any other SMTP extensions, either. This
functionality is to be added in future versions.

[rfc]: http://tools.ietf.org/search/rfc5321
[smtp2go]: http://www.smtp2go.com/articles/smtp-protocol
[mimewiki]: http://en.wikipedia.org/wiki/MIME

## Usage

### Sending an email

The object for instantiating a client capable of sending a simple email is `SmtpSimple`.
For services created with it the request type is `EmailMessage`, described in
[`EmailMessage.scala`][EmailMessage].

You can create an email using `EmailBuilder` class described in [`EmailBuilder.scala`][EmailBuilder]:

```scala
val email = EmailBuilder()
.sender("from@from.com")
.to("first@to.com", "second@to.com")
.subject("test")
.bodyLines("first line", "second line") //body is a sequence of lines
.build
```

Applying the service on the email returns `Future.Done` in case of a successful operation.
In case of failure it returns the first encountered error wrapped in a `Future`.

[EmailMessage]: src/main/scala/com/twitter/finagle/smtp/EmailMessage.scala
[EmailBuilder]: src/main/scala/com/twitter/finagle/smtp/EmailBuilder.scala

#### Greeting and session

Upon the connection the client receives server greeting.
In the beginning of the session an EHLO request is sent automatically to identify the client.
The session state is reset before every subsequent try.

### Example

The example of sending email to a local SMTP server with SmtpSimple and handling errors can be seen
in [`Example.scala`](src/main/scala/com/twitter/finagle/example/smtp/Example.scala).

### Sending independent SMTP commands

The object for instantiating an SMTP client capable of sending any command defined in *RFC5321* is `Smtp`.

For services created with it the request type is `Request`. Command classes are described in
[`Request.scala`][Request].

Replies are differentiated by groups, which are described in [`ReplyGroups.scala`][ReplyGroups].
The concrete reply types are case classes described in [`SmtpReplies.scala`][SmtpReplies].

This allows flexible error handling:

```scala
val res = service(command) onFailure {
// An error group
case ex: SyntaxErrorReply => log.error("Syntax error: %s", ex.info)
// A concrete reply
case ProcessingError(info) => log,error("Error processing request: %s", info)
// Default
case _ => log.error("Error!")
}
// Or, another way:
res handle {
...
}
```

[Request]: src/main/scala/com/twitter/finagle/smtp/Request.scala
[ReplyGroups]: src/main/scala/com/twitter/finagle/smtp/reply/ReplyGroups.scala
[SmtpReplies]: src/main/scala/com/twitter/finagle/smtp/reply/SmtpReplies.scala

#### Greeting and session

Default SMTP client only connects to the server and receives its greeting, but does not return greeting,
as some commands may be executed without it. In case of malformed greeting the service is closed.
Upon service.close() a quit command is sent automatically, if not sent earlier.
@@ -0,0 +1,85 @@
package com.twitter.finagle

import com.twitter.finagle.client.{DefaultClient, Bridge}
import com.twitter.finagle.smtp._
import com.twitter.finagle.smtp.filter.{MailFilter, HeadersFilter, DataFilter}
import com.twitter.finagle.smtp.reply._
import com.twitter.finagle.smtp.transport.SmtpTransporter
import com.twitter.util.{Time, Future}

// TODO: switch to StackClient

/**
* Implements an SMTP client. This type of client is capable of sending
* separate SMTP commands and receiving replies to them.
*/
object Smtp extends Client[Request, Reply]{

private[this] val defaultClient = DefaultClient[Request, Reply] (
name = "smtp",
endpointer = {
val bridge = Bridge[Request, UnspecifiedReply, Request, Reply](
SmtpTransporter, new SmtpClientDispatcher(_)
)
(addr, stats) => bridge(addr, stats)
})

/**
* Constructs an SMTP client.
*
* Upon closing the connection this client sends QUIT command;
* it also performs dot stuffing.
*/
override def newClient(dest: Name, label: String): ServiceFactory[Request, Reply] = {

val quitOnCloseClient = {

This comment has been minimized.

Copy link
@mosesn

mosesn Aug 11, 2014

Contributor

thinking about this more, I think this is incorrect. we don't want to close the connection when we close this service, we just want to relinquish the connection so that it can be used elsewhere. this should definitely be moved into the dispatcher.

This comment has been minimized.

Copy link
@suncelesta

suncelesta Aug 12, 2014

Author

Ok, I've tried to use wireshark with QUIT sent in dispatcher, and it shows that the request is indeed not sent. The connection is just closed without even attempting to send it. Maybe I'm just doing it wrong? It appeared to me that I should override close(deadline: Time) method - is it right?

This comment has been minimized.

Copy link
@mosesn

mosesn Aug 12, 2014

Contributor

Have you tried what I suggested about service(Quit) ensure { service.close }? Did that have the same behavior?

This comment has been minimized.

Copy link
@suncelesta

suncelesta Aug 12, 2014

Author

It's what I've recently pushed (look below), and it has correct behavior.

This comment has been minimized.

Copy link
@suncelesta

suncelesta Aug 12, 2014

Author

And in dispatcher the behavior is still incorrect when using ensure.

new ServiceFactoryProxy[Request, Reply](defaultClient.newClient(dest, label)) {
override def apply(conn: ClientConnection): Future[ServiceProxy[Request, Reply]] = {
self.apply(conn) map { service =>
val quitOnClose = new ServiceProxy[Request, Reply](service) {
override def close(deadline: Time): Future[Unit] = {
if (service.isAvailable)
service(Request.Quit).unit
else
Future.Done
} ensure {
service.close(deadline)

This comment has been minimized.

Copy link
@mosesn

mosesn Aug 8, 2014

Contributor

I don't think service(Request.Quit) before service.close(deadline) will work, because we want to close even if we can't Quit properly, but maybe ensure would be the right semantic. Can you use wireshark and check that the tcp connection is actually being torn down when you expect it to be torn down?

This comment has been minimized.

Copy link
@suncelesta

suncelesta Aug 11, 2014

Author

According to wireshark:

  1. QUIT is sent
  2. Server responds
  3. Server closes connection
  4. Client closes connection

Seems like what should be expected.

This comment has been minimized.

Copy link
@mosesn

mosesn Aug 11, 2014

Contributor

Wait, I thought you were saying that it wasn't behaving properly in the dispatcher? I think I got confused as to what was going on . . . I was trying to reply to a message you posted that said,

Yes, when testing with SMTP server stub and looking at its logs, I found out that this request is not actually sent from dispatcher, so I sought the way to enforce service(Request.Quit) before service.close(deadline)

but I couldn't find the actual message (github swallowed it maybe) so I just put it here.

Is the behavior correct now then?

This comment has been minimized.

Copy link
@suncelesta

suncelesta Aug 11, 2014

Author

Oh, sorry, I thought you were asking to check the behavior it has right
now, not with sending QUIT in the dispatcher. I'll check the latter, but I
can say that the server doesn't receive the request in that case, because
in its logs the connection is torn down by client and QUIT isn't received.

2014-08-11 18:45 GMT+04:00 Moses Nakamura notifications@github.com:

In finagle-smtp/src/main/scala/com/twitter/finagle/Smtp.scala:

  • * Constructs an SMTP client.
  • * Upon closing the connection this client sends QUIT command;
  • * it also performs dot stuffing.
  • */
  • override def newClient(dest: Name, label: String): ServiceFactory[Request, Reply] = {
  • val quitOnCloseClient = {
  •  new ServiceFactoryProxy[Request, Reply](defaultClient.newClient%28dest, label%29) {
    
  •    override def apply(conn: ClientConnection): Future[ServiceProxy[Request, Reply]] = {
    
  •      self.apply(conn) map { service =>
    
  •        val quitOnClose = new ServiceProxy[Request, Reply](service) {
    
  •          override def close(deadline: Time): Future[Unit] = {
    
  •            if (service.isAvailable)
    
  •              service(Request.Quit)
    
  •            service.close(deadline)
    

Wait, I thought you were saying that it wasn't behaving properly in the
dispatcher? I think I got confused as to what was going on . . . I was
trying to reply to a message you posted that said,

Yes, when testing with SMTP server stub and looking at its logs, I found
out that this request is not actually sent from dispatcher, so I sought the
way to enforce service(Request.Quit) before service.close(deadline)

but I couldn't find the actual message (github swallowed it maybe) so I
just put it here.

Is the behavior correct now then?


Reply to this email directly or view it on GitHub
https://github.com/twitter/finagle/pull/287/files#r16056704.

С уважением,
Валерия Дымбицкая

This comment has been minimized.

Copy link
@mosesn

mosesn Aug 11, 2014

Contributor

right--if you do service(Quit) ensure service.close(deadline) I think that will have the correct behavior, since it will wait for a response before tearing down the connection. the two tcp connections will race to tear down, but I think that's OK.

}

This comment has been minimized.

Copy link
@suncelesta

suncelesta Aug 12, 2014

Author

Here is what you suggested, I believe. Correct me if I misunderstood, I may have lost track of thought here.

This comment has been minimized.

Copy link
@mosesn

mosesn Aug 12, 2014

Contributor

Sorry, I'm having trouble reasoning about the dispatcher code without seeing it. Could you link me to a branch which has the quit in the dispatcher?

This comment has been minimized.

Copy link
@suncelesta

suncelesta Aug 13, 2014

Author

Ok, I've created a new branch. Here are the changes for you to review.

}
quitOnClose
}
}
}
}

DataFilter andThen quitOnCloseClient
}
}

/**
* Implements an SMTP client that can send an [[com.twitter.finagle.smtp.EmailMessage]].
* The application of this client's service returns [[com.twitter.util.Future.Done]]
* in case of success or the first encountered error in case of a failure.
*/
object SmtpSimple extends Client[EmailMessage, Unit] {
/**
* Constructs an SMTP client that sends a hello request
* in the beginning of the session to identify itself;
* it also copies email headers into the body of the message.
* The dot stuffing and connection closing
* behaviour is the same as in [[com.twitter.finagle.Smtp.newClient()]].
*/
override def newClient(dest: Name, label: String): ServiceFactory[EmailMessage, Unit] = {
val startHelloClient = new ServiceFactoryProxy[Request, Reply](Smtp.newClient(dest, label)) {
override def apply(conn: ClientConnection) = {
self.apply(conn) flatMap { service =>

This comment has been minimized.

Copy link
@mosesn

mosesn Aug 6, 2014

Contributor

map instead of flatMap

service(Request.Hello)

This comment has been minimized.

Copy link
@mosesn

mosesn Aug 6, 2014

Contributor

not sure this is in the right location. I don't know SMTP that well, is there any reason why we wouldn't want to start a connection with this? maybe it should just be in the dispatcher?

This comment has been minimized.

Copy link
@suncelesta

suncelesta Aug 7, 2014

Author

A hello request identifies the client, and the RFC recommends that SMTP sessions are started with it. It can be moved to the connection phase in the dispatcher, though.

This comment has been minimized.

Copy link
@mosesn

mosesn Aug 7, 2014

Contributor

Hurk, how long do SMTP sessions last? Do we start an SMTP session with a tcp connection and tear it down when we turn down the SMTP session, or do we set it up / tear it down with the request?

This comment has been minimized.

Copy link
@suncelesta

suncelesta Aug 8, 2014

Author

The first case. The session lasts until quit command is sent (and then the server closes the connection) or some connection error occurs.

Future.value(service)
}
}
}
HeadersFilter andThen MailFilter andThen startHelloClient
}
}


ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.