Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

automated checking of mail delivery logs #624

Closed
confirmordeny opened this issue Sep 24, 2012 · 53 comments
Closed

automated checking of mail delivery logs #624

confirmordeny opened this issue Sep 24, 2012 · 53 comments

Comments

@confirmordeny
Copy link

@confirmordeny confirmordeny commented Sep 24, 2012

Sometimes public bodies claim not to have received FOI requests. When requesters write to us a skilled member of them (i.e. not me because I don't know how!) writes back and says something like:

I've checked the mail delivery logs for
http://www.whatdotheyknow.com/request/[...] and I can
confirm that both your messages were successfully delivered to [public authority]'s mail servers.

I just wonder whether it might be possible to show on the request page a little green light or something that shows each outward message was successful delivered. Perhaps it could also show the date and time of delivery. I have no idea how difficult it is.

It is far from high priority. I feel it would add something to the trustworthiness of the record because people could think to themselves "yes they really did received this message"

@Flupsy

This comment has been minimized.

Copy link

@Flupsy Flupsy commented Sep 24, 2012

It's a great idea, but unfortunately it's impossible for us to say that the message has been delivered to the authority itself. The best we could say is that 'this message has been delivered to the authority, or possibly an organisation that handles mail for them'. Lots of councils use people like Messagelabs for content scanning, and we don't know what happens to the email once it's passed to them.

@frabcus

This comment has been minimized.

Copy link

@frabcus frabcus commented Sep 24, 2012

John, should be quite easy to do, as the Exim logs are already loaded
into the database, it is a case of somebody who understands mail
parsing them and making a nice display.

Ian, we consider in a social/political (rather than technical!)
context that delivery to the MX server for their domain is delivery.

Any failure to deliver after that point is the authority's internal
responsibility. That they might outsource (part of) that work to
somebody else is irrelevant.

In the UK, in practice, this works quite well.

We often give the delivery log file lines from Exim to their technical
people, and they find out why it wasn't processed, and what went
wrong. I think quite a few authorities whitelisted our domain as a
result - certainly we often ask them to!

Francis

On Mon, Sep 24, 2012 at 12:28:34AM -0700, Ian Chard wrote:

It's a great idea, but unfortunately it's impossible for us to say that the message has been delivered to the authority itself. The best we could say is that 'this message has been delivered to the authority, or possibly an organisation that handles mail for them'. Lots of councils use people like Messagelabs for content scanning, and we don't know what happens to the email once it's passed to them.


Reply to this email directly or view it on GitHub:
#624 (comment)

@hsenag

This comment has been minimized.

Copy link
Collaborator

@hsenag hsenag commented Sep 24, 2012

In the common case, the message is either delivered straight away by WDTK's outgoing mail server, or delivery fails and a bounce appears on the request thread. So in a sense the lack of a bounce is the green light.

Trickier cases come when we're occasionally relaying through another host for some reason - usually because of delivery problems from WDTK direct to that authority in the past.

In that case WDTK does deliver the email somewhere, but it hasn't necessarily got as far as the advertised MX for the domain, and we can't ever be absolutely certain it got to the authority's own mail server.

I guess a nice display would consist of "successfully delivered to [mail server] at [time]" associated with a particular message, and then someone who vaguely understood mail would be able to confirm whether it looks like it got to the authority or not, without needing to ask us. The key bit of "technical" knowledge here is being able to look at the recipient mail server and rule out the very occasional case where the message might have got lost in a relay we chose to use.

@hsenag

This comment has been minimized.

Copy link
Collaborator

@hsenag hsenag commented Oct 21, 2014

The other tricky case is where an authority email address is out of date to the point of not being received by the authority at all [if it's the "wrong" address but still within the authority that's fine, at least under UK law]. This case is most common for small authorities like schools and parish councils.

A simple green tick is quite attractive for its simplicity. An embellishment might be to hide it once there's a response from the authority. Another embellishment might be to have a click through to more detailed logs - perhaps the raw exim logs with the request address redacted.

@crowbot crowbot added the 1 - new label Oct 23, 2014
@crowbot crowbot added 2 - contender and removed 1 - new labels Nov 11, 2014
@RichardTaylor

This comment has been minimized.

Copy link

@RichardTaylor RichardTaylor commented Nov 18, 2014

I was discussing this recently and some ideas included;

  • A badge / stamp making clear that making a request via an Alaveteli service is equivalent to having "proof of posting" for a letter. We could have some kind of graphical "seal/stamp" confirming the message has been sent.
  • I think we can go further and when we have a "250 OK" response from the mail servers an authority is using we should be able to update the thread with something like "Delivery Confirmed"; with a link to some text adding caveats as mentioned above eg. the request might still be in spam trap.
@RichardTaylor

This comment has been minimized.

Copy link

@RichardTaylor RichardTaylor commented Nov 25, 2015

ICO decision notice FS50559082 concluded, after considering message delivery evidence from WhatDoTheyKnow

The Commissioner considers that there is sufficient evidence to show that the Cabinet Office had received the request on the date it was sent by the complainant. As it did not issue its refusal notice within 20 working days, it breached section 17(5) of the Act.

https://www.whatdotheyknow.com/request/appointment_of_qualified_persons#comment-64694

@crowbot crowbot added the t:design label Dec 11, 2015
@zarino

This comment has been minimized.

Copy link
Member

@zarino zarino commented Jan 5, 2016

How big a deal do we want to make of this? A very subtle solution might be:

unknown-status

(where the question mark is a link to an FAQ item about how we track delivery, and why we've said "Sent" instead of "Delivered" in this case)

And then when it’s accepted by the mail server:

delivered

(where the tick reveals a popup or something showing what we mean by "delivered" perhaps even with some geeky mail server log for people who want to use it as some form of evidence)

Just my first thought. Any feedback?

@crowbot

This comment has been minimized.

Copy link
Member

@crowbot crowbot commented Jan 5, 2016

I'm in favour of having the mail server log available, but hidden unless you actively seek it out - I think it would be good to have an indication in the 'delivered' mode that there is more to see - like the question mark in 'sent' mode. This could link to an explanation of the complexities of mail delivery, and an appropriately redacted version of the delivery log. Maybe we should show the requester an uncensored version that they could forward to the authority.

@zarino

This comment has been minimized.

Copy link
Member

@zarino zarino commented Jan 5, 2016

@crowbot: What are the situations in which somebody would seek out this proof of an email being delivered?

@crowbot

This comment has been minimized.

Copy link
Member

@crowbot crowbot commented Jan 5, 2016

The authority claiming that they never received the request.

@zarino

This comment has been minimized.

Copy link
Member

@zarino zarino commented Jan 5, 2016

Will that ever be before the 20 working days has elapsed? And will it ever be after the authority has sent their first reply?

@zarino

This comment has been minimized.

Copy link
Member

@zarino zarino commented Jan 5, 2016

(I'm wondering whether it can be shown more intelligently – eg: inside the "status" line we already put above the opening request)

@crowbot

This comment has been minimized.

Copy link
Member

@crowbot crowbot commented Jan 5, 2016

I think the answer to the first question could be yes - sometimes the team gets emails from authorities saying - "I've just browsed the site, and there are a load of requests we've never received" - and these might include relatively new requests. Less likely but still possible that the authority could contest delivery in or after their first response e.g. they could reply using the 'respond to this request' option and say "we never received it by email".

@zarino

This comment has been minimized.

Copy link
Member

@zarino zarino commented Jan 5, 2016

Good to know.

Shall I do a quick prototype of this in a branch, so we can see how it feels?

And if so, do you have any specific wording I should use?

@crowbot

This comment has been minimized.

Copy link
Member

@crowbot crowbot commented Jan 5, 2016

Yes, that sounds great - I don't have any specific wording in mind at the moment.

@RichardTaylor

This comment has been minimized.

Copy link

@RichardTaylor RichardTaylor commented Jan 5, 2016

Maybe we should show the requester an uncensored version that they could forward to the authority.

Even requestors should have the request email address redacted. A naughty requestor could fake a response if they had the address, damaging the credibility of the service.

What are the situations in which somebody would seek out this proof of an email being delivered?

I think the ~"delivery confirmed" message should be aimed at all those browsing the site; it's something which might encourage people to choose to use the service rather than make a request in private.

@hsenag

This comment has been minimized.

Copy link
Collaborator

@hsenag hsenag commented Jan 5, 2016

It'd also be good to make the raw logs available (with the checksum part of the email address redacted) so that requesters can send them to authorities or the ICO directly.

zarino added a commit to mysociety/alavetelitheme that referenced this issue Jan 6, 2016
zarino added a commit that referenced this issue Jan 6, 2016
@crowbot crowbot added 4 - now and removed 2 - contender labels Jan 6, 2016
@garethrees

This comment has been minimized.

Copy link
Member

@garethrees garethrees commented May 12, 2016

The ability to view Mail Server Logs has been deployed (#3209, for admins only) with the intention of removing the admin-only filter early this week.

Just rolled this out to both admins and request owners (#3248). Not a massive amount of value in showing somewhat sensitive (and redacted) logs to non-request owners, but will show them the delivery status when that's available.

@kingqueen3065

This comment has been minimized.

Copy link
Collaborator

@kingqueen3065 kingqueen3065 commented May 14, 2016

This is a major step forward and an important breakthrough. I have a signficant concern, though.

If requesters have access to the full request email address of their requests, including the checksum, there is no doubt that some of them will abuse it by sending mail appearing to come from the public authority. We inevitably have a number of vexatious users who would make use of this to cause mischief.

Also, some users will doubtless manage to get confused and use the request addresses to send allsorts e.g. immigration concerns to that request address. It's amazing how confused some people with limited technical ability or limited English language can get - for example, we regularly get requests to change our record of public authorities' email addesses to somebody's personal address as an attempt to raise contact the authority over immigration and other matters. Though this is of less concern than the vexatious one.

In my opinion, the unredacted request address must only be available to the authority responding to each request, and to admins and developers. Otherwise there will be all sorts of problems and the integrity of the communications on our website ostensibly sent by public authorities will be in doubt.

The "release mail log" feature is incredibly important and useful; we frequently confirm that authorities' email servers have received request emails, and it's a great step forward, but if we're providing the log file to anybody other than the authority, or publishing it as an annotation on the request page, we always redact the checksum - request-XXXXXX-(redacted)@whatdotheyknow.com - for this reason.

@garethrees

This comment has been minimized.

Copy link
Member

@garethrees garethrees commented May 20, 2016

Should write some documentation about this for alaveteli.org admin guides. Will probably wait until we've got the delivery status parsing done. Could also do with better user-facing documentation about what they are, though this might become clearer with the delivery status parsing.

@garethrees

This comment has been minimized.

Copy link
Member

@garethrees garethrees commented Jun 7, 2016

Once #3214 is merged, we need to design and implement styles for the available delivery statuses (sent, delivered, failed) when showing an outgoing message's mail server logs. Probably need a little collaboration with the designers to get the variables in the right places. Off the top of my head we'll want:

<%# Print a message like "This message has been delivered successfully" %>
<p class="delivery-status-explanation"><%= @delivery_status.humanize %></p>
<%# @delivery_status.humanize # => "This message has been delivered successfully" %>

<%# Use the "simple" delivery status for the styling and icons. Returns a Symbol %>
<%# Symbol to String conversion handled automatically %>
<%# @delivery_status.simple # => :delivered %>
<%# @delivery_status.simple.to_s # => 'delivered' %>
<i class="icon icon_<%= @delivery_status.simple %>"></i>

<%# Use the "simple" delivery status for display %>
<%# You must manually convert to String so that humanize can be called %>
<%# @delivery_status.simple.to_s.humanize # => 'Delivered' %>
<strong class="icon icon_<%= @delivery_status.simple %>">
  <%= @delivery_status.simple.to_s.humanize %>
</strong>

Here's an example of a diff you can apply to make an OutgoingMessage return some sort of delivery status:

diff --git a/app/models/outgoing_message.rb b/app/models/outgoing_message.rb
index 5a14ea2..32fb701 100644
--- a/app/models/outgoing_message.rb
+++ b/app/models/outgoing_message.rb
@@ -251,6 +251,12 @@ class OutgoingMessage < ActiveRecord::Base
   #
   # Returns an Array.
   def mail_server_logs
+    log_lines = <<-EOF.strip_heredoc.split("\n")
+    2015-10-30 19:24:16 [17814] 1ZsFHb-0004dK-SM <= request-123-abc987@example.net U=alaveteli P=local S=2252 id=ogm-14+537f69734b97c-1ebd@localhost T="FOI Request about stuff" from <request-123-abc987@example.net> for authority@example.com authority@example.com
+    2015-10-30 19:24:16 [17817] 1ZsFHb-0004dK-SM => authority@example.com F=<request-123-abc987@example.net> P=<request-123-abc987@example.net> R=dnslookup T=remote_smtp S=2297 H=cluster2.gsi.messagelabs.com [127.0.0.1]:25 X=TLS1.2:DHE_RSA_AES_128_CBC_SHA1:128 CV=no DN="C=US,ST=California,L=Mountain View,O=Symantec Corporation,OU=Symantec.cloud,CN=mail221.messagelabs.com" C="250 ok 1446233056 qp 26062 server-4.tower-221.messagelabs.com!1446233056!7679409!1" QT=1s DT=0s
+    EOF
+    logs = log_lines.map { |line| MailServerLog.new(:line => line, :info_request_id => self.info_request_id) }
+    return logs
     case AlaveteliConfiguration.mta_log_type.to_sym
     when :exim
       exim_mail_server_logs
OutgoingMessage.last.delivery_status
# => #<MailServerLog::PostfixDeliveryStatus:0x007dcb2f0 @status=:sent>

If you want a specific delivery status, you can forcefully return one:

diff --git a/app/models/outgoing_message.rb b/app/models/outgoing_message.rb
index 5a14ea2..c3c6dea 100644
--- a/app/models/outgoing_message.rb
+++ b/app/models/outgoing_message.rb
@@ -262,6 +262,7 @@ class OutgoingMessage < ActiveRecord::Base
   end

   def delivery_status
+    return ::MailServerLog::EximDeliveryStatus.new(:normal_message_delivery)
     mail_server_logs.
       map { |log| log.line(:decorate => true).delivery_status }.
         compact.

You can pick any delivery status from

DELIVERED_FLAGS = [
:normal_message_delivery,
:additional_address_in_same_delivery,
:cutthrough_message_delivery
].freeze
SENT_FLAGS = [
:message_arrival,
:delivery_deferred_temporary_problem
].freeze
FAILED_FLAGS = [
:bounce_arrival,
:delivery_suppressed_by_N,
:delivery_failed_address_bounced
].freeze

@garethrees

This comment has been minimized.

Copy link
Member

@garethrees garethrees commented Jun 17, 2016

@garethrees

This comment has been minimized.

Copy link
Member

@garethrees garethrees commented Jun 21, 2016

@crowbot is looking at postfix in #3337

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.