In this tutorial we will be learning how to make a python program that can read emails and respond based on the content. Once you have the fundamentals of the program, you can write your own functions to do anything. I will provide a 2 examples: a weather bot and a remote commands/shell script executor (useful on your own machine rather than repl.it). Sorry for the tutorial being so long. If you just want to get to the bot, skip the "IMAP in Python Basics" section and the "SMTP in Python Basics" sections.
IMAP stands for Internet Mail Access Protocol. Almost all email clients, even web based ones, login to your email providers email server using IMAP. With IMAP there are folders to organize emails and emails stay on the server unless explicitly deleted.
SMTP stands for Simple Mail Transfer Protocol. It is the standard protocol used for sending emails across the internet. SMTP is used to both send and receive emails, but we will only be sending.
To implement IMAP we will be using the third-party module imapclient
(docs here).
Most servers can be found with a simple google search of "your provider imap settings", but here are the most common ones:
Gmail: imap.gmail.com
Outlook.com/Hotmail.com: imap-mail.outlook.com
Yahoo.com: imap.mail.yahoo.com
AT&T: imap.mail.att.net
Comcast: imap.comcast.net
Verizon: incoming.verizon.net
Now you can connect to the server, the imapclient
module makes this easy. All you have to do is initialize an imapclient.IMAPClient()
object using the server name, and it will connect to it on the default IMAP port using SSL. For example:
import imapclient
i = imapclient.IMAPClient('imap-mail.outlook.com') # i is easy to type
Logging in is very simple.
i.login('bob@example.com', 'mySuperSecretPassw0rd')
The i.login()
function is pretty self-explanatory, it logs in using the email and password provided.
Next we need to select a folder. i.list_folders()
shows the available folders. You should choose the one you want, usually "INBOX".
i.select_folder('INBOX')
You can also, for debugging, pass in the keyword argument readonly=True
to select_folder() to make the server not mark the messages you fetch as read.
Now we can search for a message. There are many search criteria, here is a full list. Some of the most common ones are 'ALL'
which selects all message, 'UNSEEN'
which gets all unread messages. The search flags behave differently which each email server, so you should first experiment with them in the shell. You can search through the messages in the folder using i.search(['criteria'])
. To get a unread messages, use the following code:
i.search(['UNSEEN'])
This will not return the emails themselves, but a list of unique IDs or UIDs, for example [40032, 40033, 40034]
which we are about to use.
You can also use a special method if you are using Gmail, i.gmail_search()
which behaves like the search box in Gmail.
If your search matches lots of messages, you can get an imaplib.erorr: got more than 10000 bytes
. This can be fixed by the following code:
import imaplib
imaplib._MAXLINE = 1000000
This sets the limit to 10,000,000 bytes instead of 10,000.
When you fetch an email, you download it from the server. Unless you are using readonly=True
, when you fetch an email it will mark it as read. You fetch an email like this:
rawmsgs = i.fetch(uids, ['BODY[]']) # uids is the uids returned by search()
The object returned by fetch is complicated and hard to parse, so we will be using a second third-party module, pyzmail
to parse them. ** Important: If you are installing pyzmail using pip on python 3.6 or above, you need to install pyzmail36
instead of pymail
or you will get an error in pip. **
This is how you parse a message with pyzmail:
import pyzmail
msg = pyzmail.PyzMessage.factory(rawmsgs[40041]['BODY[]']) # Be sure to change the uid number
Using this object you can get a lot of information on the message.
msg.get_subject()
msg.get_addresses('from')
msg.get_addresses('to')
msg.get_addresses('cc')
Sample Output:
Thanks!
[('Alice Doe', 'alice@example.com')]
[('Bob Smith', 'bob@example.com')]
[]
These methods get the subject and addresses the message was sent to.
msg.text_part != None
msg.text_part.get_payload().decode(msg.text_part.charset)
msg.html_part != None
msg.html_part.get_payload().decode(msg.html_part.charset)
Sample Output:
True
'Thanks for buying lunch yesterday. \r\n\r\n-Alice\r\n'
True
'<div style="font-family: courier;">Thanks for buying lunch yesterday. <br><br>-Alice</div>'
Email messages can have 2 parts: a text part and an HTML part. In pyzmail, they
will either be None
if it doesn't exist or if it does exist it will have
a get_payload()
method which returns bytes
which can be decoded
using .decode()
with the charset stored in either msg.text_part.charset
or msg.html_part.charset
. The text part is plaintext, the HTML part is
HTML to be rendered for the user.
SMTP is similar to IMAP, but doesn't have as many commands. To implement SMTP,
we will be using the python built in library, smtplib
.
You should first find your providers SMTP settings by finding them in the list
below or searching your provider smtp settings.
Gmail: imap.gmail.com
Outlook.com/Hotmail.com: imap-mail.outlook.com
Yahoo.com: imap.mail.yahoo.com
AT&T: imap.mail.att.net
Comcast: imap.comcast.net
Verizon: incoming.verizon.net
Once you have your settings, you can connect to the server by initializing a new
smtplib.SMTP()
object and starting TLS:
s = smtplib.SMTP('smtp.example.com', 587)
s.starttls()
s.ehlo()
If you get an error while connecting or if your email provider is listed above as port 465 then your email provider may not support TLS on port 587. In this case, you should connect using SSL to port 465 like this:
s = smtplib.SMTP_SSL('smtp.example.com', 465)
s.ehlo()
Logging in is similar to IMAP, you call the login()
function of your SMTP
connection, using your email and password: Be careful about leaving passwords
in your code!
s.login('bob@example.com', 'mySuperSecretPassw0rd')
To send emails, I have found that the sendmail()
function doesn't really
work as it can mess with email headers, instead we should use the python
built-in module email
's email.message.EmailMessage()
class which
makes settings the headers really easy and has many advanced capabilities which
are listed in the docs.
Here is an example of how to send a text message:
text = """Dear Alice,
You are welcome. Think nothing of it!
-Bob
"""
msg = email.message.EmailMessage()
msg['from'] = 'bob@example.com'
msg["to"] = 'alice@example.com'
msg["Subject"] = "Re: Thanks! "
msg.set_content(text)
res = s.send_message(msg)
In most email providers, though not required, prefacing the subject with Re:
denotes a reply. In this example, Re: Thanks! would show up in the same
thread as the original Thanks! message in both email clients. To return
value of the send_message()
function is a dictionary of the addresses
to which sending failed. If successful, it should return {}
.
Using our knowledge, we are going to write the bot. First thing we need to do
is store the configuration information. For repl.it, we will use input()
but on your own computer you can hard-code values (not passwords! ).
Add a new file in repl.it called config.py
and add the following:
import getpass
radr = input("Adresses to log in to") # address to check and send from
imapserver = input("IMAP server domain: ") # imap server for account
smtpserver = input("SMTP server domain: ") # smtp server for account
smtpserverport = input("SMTP Server port [587]: ") # smtp server port for starttls
if not smtpserverport or smtpserverport == "":
smtpserverport = 587
pwd = getpass.getpass("Account password: ") # password for account encoded with base64.b64encode
sadr = input("Trusted addresses to receive from: ") # address to receive commands from
check_freq = 5
Here we set the config values to be used by the main program. For security, we will set a trusted email that is the only one that commands will be accepted from. This is so that no random people can email us commands.
from config import *
def imap_init():
"""
Initialize IMAP connection
"""
print("Initializing IMAP... ", end='')
global i
i = imapclient.IMAPClient(imapserver)
c = i.login(radr, pwd)
i.select_folder("INBOX")
print("Done. ")
Here we initialize the imap connection by import our config file
(using import *
is usually a bad idea because you don't know where things
came from but we are just importing variables so it is okay).
We also define i using global i
so that it is available to the rest of our
program. We also login to the server and select the "INBOX" folder.
def smtp_init():
"""
Initialize SMTP connection
"""
print("Initializing SMTP...")
global s
s = smtplib.SMTP(smtpserver, smtpserverport)
c = s.starttls()[0] # The returned status code
if c is not 220:
raise Exception('Starting tls failed: ' + str(c))
c = s.login(radr, pwd)[0]
if c is not 235:
raise Exception('SMTP login failed: ' + str(c))
print("Done. ")
In this block, we initialize the SMTP connection. We use the same global
technique as with IMAP and connect to the SMTP server and login, but we also
check the status codes returned by the server to make sure everything was
successful (we can't do this with IMAP).
The next step is getting any unread emails.
def get_unread():
"""
Fetch unread emails
"""
uids = i.search(['UNSEEN'])
if not uids:
return None
else:
print("Found %s unreads" % len(uids))
return i.fetch(uids, ['BODY[]', 'FLAGS'])
Here we define a function to get unread emails. This function searches the IMAP
object for any unread emails. It returns None
if it didn't find any, or
if it did, it fetches them from the server and returns them.
For our bot to work, we have to define some actions for it. To do this, we will
define functions in config.py
and add them to a commands dict
which
will map the command names to the functions. The functions will take 1
argument: the message split by lines. For now, we will make a command that
will return "Hello, World! " every time no matter the message content:
def hello_world(lines):
return "Hello, World! "
commands = {"hello" : hello_world}
Now we will analyze the message and determine what to do about it.
def analyze_msg(raws, a):
"""
Analyze message. Determine if sender and command are valid.
Return values:
None: Sender is invalid or no text part
False: Invalid command
Otherwise:
Array: message split by lines
:type raws: dict
"""
print("Analyzing message with uid " + str(a))
msg = pm.factory(raws[a][b'BODY[]'])
frm = msg.get_addresses('from')
if frm[0][1] != sadr:
print("Unread is from %s <%s> skipping" % (frm[0][0],
frm[0][1]))
return None
global subject
if not subject.startswith("Re"):
subject = "Re: " + msg.get_subject()
print("subject is", subject)
if msg.text_part is None:
print("No text part, cannot parse")
return None
text = msg.text_part.get_payload().decode(msg.text_part.charset)
cmds = text.replace('\r', '').split('\n') # Remove any \r and split on \n
if cmds[0] not in commands:
print("Command %s is not in commands" % cmds[0])
return False
else:
return cmds
Let's break this down. First we initialize the message object with the data we
are given. Next, we make sure it is from our trusted address. Next we set the
subject to start with Re: so that it shows up as a reply in the email thread.
Next, we make sure we have a text_part
to parse and split it by lines.
We check that the requested command is a valid command, and if so, return the
line of the message.
Defining a mail function will be useful so that we can see in the logs what was mailed to the user, and it will make the sending mail process as easy as passing the function the mail text.
def mail(text):
"""
Print an email to console, then send it
"""
print("This email will be sent: ")
print(text)
msg = email.message.EmailMessage()
global subject
msg["from"] = radr
msg["to"] = sadr
msg["Subject"] = subject
msg.set_content(text)
res = s.send_message(msg)
print("Sent, res is", res)
We will write the loop our bot goes through as it waits for a message to process. We want our bot to run until we interrupt it, so we will put it in an infinite loop.
imap_init()
smtp_init()
while True: # Main loop
try:
print() # Blank line for clarity
msgs = get_unread()
while msgs is None:
time.sleep(check_freq)
msgs = get_unread()
for a in msgs.keys():
if type(a) is not int:
continue
cmds = analyze_msg(msgs, a)
if cmds is None:
continue
elif cmds is False: # Invalid Command
t = "The command is invalid. The commands are: \n"
for l in commands.keys():
t = t + str(l) + "\n"
mail(t)
continue
else:
print("Command received: \n%s" % cmds)
r = commands[cmds[0]](cmds)
mail(str(r))
print("Command successfully completed! ")
except KeyboardInterrupt:
i.logout()
s.quit()
break
except OSError:
imap_init()
continue
except smtplib.SMTPServerDisconnected:
smtp_init()
continue
finally:
i.logout()
s.quit()
This is a long code block, so let's break it down. First we check for unread
messages and if there are none, then we wil get into a loop waiting for one.
Next we loop over each of the unreads we have retrieved and analyze them. (there
are a lot of loops in this portion). We then check the value returned by analyze_message()
and if it was False
meaning an invalid comand we send the user a helpful
message detailing the available comands. If it was None
meaning it is
not from the trusted sender or has an invalid text_part
we will skip it
and continue the loop.
Here we check the return value from analyze_msg()
. If it is None (meaning
it is not from the trusted sender or doesn't have a text part) we skip it and
continue to the next message. If it is False, meaning it is an invalid command,
we helpfully send the user a message reminding them of the available commands.
Otherwise, we assume we got the text to fun so we run the corresponding command.
When it finishes, we send back the result and print a message to the console.
The bot is complete!
Now we can run the bot and test it! If you run the bot and send a message to
the email it is connected to with the first line being hello
then it
should reply back "Hello, World! ". If you get any error, make sure you followed
each step and if you think the error is on my part, let me know in the comments
and I will fix it.
Here I will provide you with one of two example command: a program that gets the weather using the openweathermap.org API for your local area and emails it back to you!
Before you can use this program, you need to find a couple of values. To find
your city's id, search for it at https://openweathermap.org/find and click on it
in the results. The city id is the string of numbers at the end of the URL.
To get your API key, follow the steps here: https://openweathermap.org/appid.
This requires you to create a free account and wait one hour for you API key to
be activated. Keep in mind the limit for calls with a free account is one per
10 minutes, so don't check too often or your account could be suspended. Now
that you have your values, replace them in the code below. Also, we have one
third-party dependency: the requests
module. Don't forget to install it
if you haven't already.
import requests
from json.decoder import JSONDecodeError
def weather(lines):
city_id = str(5391959)
url = "https://api.openweathermap.org/data/2.5/forecast?id="
api_key = "your api key here"
res = requests.get(url + city_id + "&APPID=" + api_key + "&mode=json")
if res.status_code != 200:
return "Oops, code was " + code
try:
j = res.json()
except JSONDecodeError:
return "JSON decode error: \n" + res.text
try:
main = j["city"]["list"][0]["weather"][0]["main"]
description = j["city"]["list"][0]["weather"][0]["description"]
except Exception as e:
return str(e) + "\n" + r.text
return "The weather is: \n%s: %s" % (main, description)
Let's break this down. First we define our config values and get the API page using them. Then, we make sure the request was successful (if the status code was 200) and send a message back if it wasn't. Next we attempt to decode the JSON returned by the API and send back if we can't. Finally, we get the weather and the description of it and catch any error that occurs from it (usually the key not existing because of an error with the call) and send back the error and full content of the request so that we can diagnose the error. Finally, if all was successful, we return the weather to the user!
This command won't be useful on repl.it, but rather on your own server. This is really two commands: one that will run a single command, and one that will write the message contents to a file on disk and execute it. (useful if you want to run commands in the same working directory or with variables because if you only run one command at a time, the shell will be reset each time).
import subprocess as sub
def exec_cmd(cmds):
param = cmds[2]
"""exec_cmd: executes a command with subprocess
"""
try:
p = sub.run(param, shell=True, timeout=20, stdout=sub.PIPE, stderr=sub.PIPE)
except sub.TimeoutExpired:
return "Command timed out. "
return '''Command exited with code %s
Stdout:
%s
Stderr:
%s
''' % (p.returncode, p.stdout.decode(), p.stderr.decode())
This command isn't super complicated. We take the command to be executed as the
second line and run it using subprocess
with a timeout of 20. If the
timeout expires, we inform the user. Otherwise, we send the return code,
STDOUT, and STDERR.
def runscript(lines):
"""
Writes input to disk and executes it. Returns any errors.
IMPORTANT:
Before running, make sure file 'script' exists and is executable.
"""
try:
f = open("script", "w")
count = 0
for i in lines:
if count is 0 or i is 1:
count = count + 1
continue
f.write(i + "\n")
count = count + 1
f.close()
p = sub.run("./script", shell=True, timeout=timeout, stdout=sub.PIPE, stderr=sub.PIPE)
r = """Script finished!
Return code: %s
Stdout:
%s
Stderr:
%s
""" % (p.returncode, p.stdout.decode(), p.stderr.decode())
except Exception as e:
return "Error: \n" + str(e)
return r
This code is extremely similar to the single command runner code, except for it
loops through the lines for the message (ignoring lines 1 and 2) and writes them
to a file named script
which, as mentioned in the docstring, should exist
and be executable before this command runs.
To add these commands to the bot, and to define you own, paste the functions
into emailcmd_config.py
and update the the commands
dictionary at
the bottom by putting the key as the alias for the command and the value as the
function object.
commands = {"weather":weather, "exec":exec_cmd, "script":runscript}