Link news stories and articles to requests

[garethrees]

A subset of #34.
Fixes https://github.com/mysociety/transparency-adessium/issues/26.

---

We want to build a culture of fact-based journalism.

One way of doing this is for journalists to show their sources, especially in the case of public information like FOI requests. We know it's a struggle to get them to do this at the story end, but we think there could be milage in linking the other way – from the source to the story.

Benefits for Alaveteli maintainers:

• help track and show impact for credibility and funding purposes

Benefits for Pro users:

• Give them a little free publicity once the data is published
• Encourage them to actually publish their requests (mysociety/alaveteli-professional#385), due to 1

Benefits for free users:

• Increase motivation, knowing that their seemingly small acts can have bigger impact if their requests get picked up by news publications.

Naming

We're going to call these "Citations" – they're similar to an "annotation", but with more structured data.

Who can add citations?

For the initial launch, we're only going to allow the following users to create citations:

• Admins
• Pros
• Request owners, on their own requests

There's a little more risk of abuse from allowing request owners to add citations, but I think in general we trust our users and their mass will help this idea pick up more than if we only initially allowed Admins and Pros to add citations.

We can later think about allowing all users to add citations, possibly through a "suggestion" form similar to PublicBodyChangeRequest.

The blank slate

The request page sidebar will be the kicking-off point for Citations. We think its important to give these some prime space since they're a new concept.

We'll add a new .sidebar__section container with a little explainer, and a link to add a new citation. We need to work out the wording here, but I'm thinking along the lines of:

IN THE NEWS

Has this request been mentioned in a news article or academic paper?

Let us know

We'll disable the link to the form unless the user has permission to add citations.

The form can be pretty simple. At a minimum we'll record the source URL.

We'll aim to store the "type" of citation (whether its a news story, academic paper or other). This helps users understand the type of citations we're after. This isn't critical though.

We may also want to add a text field for a human-readable title. We could try to automatically populate this with the <title> tag contents from the source URL, but it depends how much time we have left.

On submission, we just show a flash message saying thanks.

We should reject duplicate source URLs.

Displaying and adding more citations

We'll show the most recent 3-5 citations in the sidebar. We could prioritise "news"-type citations in the future, but probably we'll just go for most recent initially.

Its not critical for launch if we're really pushed, but the aim should be to have a page listing all citations, with further details than we can show in the sidebar.

Clicking a citation link, or "more" should take you to the citations index (/request/:url_name/citations).

Here we'll show a little request metadata, and expand citations, showing who added it (for accountability) and when. The citations here can then link off-site to the source URL.

Editing and deletion

We want to avoid getting bogged down by figuring out exact permissions of who can edit/delete citations in various contexts (e.g. "Can a Pro delete a citation on their request that got added by an Admin?"), so for this first iteration we'll make editing and deletion pretty limited.

We should only allow deletion within 30 minutes of creation by the citation author. After that they'll have to get in touch with the admin team, and developers will have to remove/edit through the console.

Batch requests

We'll aim to mirror the behaviour of citations on Batch requests – adding an "In the news" section to the sidebar.

Citations will be recorded against the InfoRequestBatch.

When collecting citations to show alongside a batch, we'll get both the citations directly added to the batch, plus citations added to individual requests within the batch.

On a request page that belongs to a batch, we'll collect citations added to that particular request, plus citations added to the parent batch.

Not essential, but when adding a citation to an individual request in a batch, we could add a tick box that allows the citation to get added to the batch record instead of the individual request:

The data model

We decided that tags aren't that useful for this, as they only store one value.

Instead we'll create a Citation model:

# Citation
#   source_url: string    # The URL to the news story etc
#   user_id: int          # The user who added the citation
#   type: string          # News story / Academic Paper / other
#   created_at: datetime
#   updated_at: datetime

There's a lot of overlap with the current Comment model, so we'll aim to extract the common bits (visibility, reporting, admin columns) out in to an Annotation abstract class, with the more specific details (e.g. validations) in subclasses.

The association will have to be polymorphic, so that a Citation can get added to an InfoRequest or an InfoRequestBatch.

We'll start off by creating Citation first, if possible, so that at least we have that if we have to abandon the abstract class idea.

Human title

When listing citations, it would be nice to show something more human friendly than just the URL. There are a couple of options for this, depending on the time available once we've got the core features done.

I think the default should be to just show the URL, less the protocol and www.

So for a source URL of https://www.bbc.co.uk/news/uk-scotland-highlands-islands-50508800 we'd show: bbc.co.uk/news/uk-scotland-highlands-islands-50508800.

[RichardTaylor]

Fantastic.

• What about a "latest requests in the news" page, and/or section on the front-page?

• A public body page could have a side-bar section listing any latest requests to that body with citations too.

• Maybe a user page as well - could highlight requests by that user which have been cited.

• This has got me considering if it might be desirable to designate a set of already made requests as a batch - so citations can be made against them.

• It should be possible with some manual work to "seed" this feature by searching existing annotations for links to news sites. Perhaps users could be encouraged to help with this.

• Risks associated with allowing all users to propose citations could be limited if the set of websites their citations can link to were limited to eg. domains already in existing citations or a list of major publisher domains - to prevent links out to arbitrary spammy sites.

[garethrees]

Fantastic.

Thanks 🎉

We've set a limited time budget for the initial implementation of this, so we're unlikely to extend the scope past what's described in the issue body above. All great ideas though, and will tackle separately in the future. Just focusing on laying the groundwork first.

[MyfanwyNixon]

I also have some feature creep ;) which I appreciate there isn't time to implement if it's not already been considered.

• Will admin be able to access a list of citations?
• Even better (for my comms purposes) would be an alert every time one is added.
• Will citations be one of the search options, eg would theguardian.com bring up any requests with that partial URL attached?

[garethrees]

Will admin be able to access a list of citations?

Not yet, but yes, we'd like to make a public list

Even better (for my comms purposes) would be an alert every time one is added.

Not yet, but easy to generate a manual report for now.

Will citations be one of the search options, eg would theguardian.com bring up any requests with that partial URL attached?

Not yet, but interesting idea.

[garethrees]

Personally while I see it’s the start of a new feature, it’s not got features which would encourage me to use it eg. giving greater prominence to the request once it’s been cited.

From volunteer catch up call notes

[MyfanwyNixon]

Further suggestions for scope increases, if we ever get the resource:

• Use whatever technology Google Docs does to grab the story's title instead of just displaying the URL (or more simply, as the user to input it)
• Make this area look more prominent than it does now - it is easy to miss.
  
• I'm actually not sure 'citation' is the best word for this feature; it feels back to front, ie, the request is the thing that the news story cites, not the other way around.

[RichardTaylor]

It appears that this feature is live on WhatDoTheyKnow but only admins (and requestors?) can add links?

Suggestions / observations:

• Make clear who the ability to add links is being offered to and why.
• Adding a link led to confusion for me as it didn't immediately appear on the request page for a non-logged in user. Adding cache busting parameters to the URL (adding eg. ?odmflsd=dsnfnds) made the links appear. Slow caches / caches which don't update as the content update are surprising and confusing.
• Consider the text: "Has this request been referenced in a news article or academic paper? Let us know:" That's a high bar, it perhaps requires the article to have explicitly referenced the request on WhatDoTheyKnow. Currently journalists often don't explicitly cite their sources and instead use phrases such as "documents seen by [name of organisation]", "documents obtained by [name of organisation]", or ~"documents obtained via Freedom of Information laws", the latter notably not saying who made the request.
• It would be good to give extra credit to journalists etc. who do actually cite their source.

[RichardTaylor]

Requests in the news should be included in alerts.

Perhaps significantly newsworthy requests should be in everyone's alerts by default?

Care would need to be taken to remain impartial.

[garethrees]

Closing this because the described ticket is complete. For enhancements to the core functionality other tickets can be opened for consideration.

[RichardTaylor]

human-readable title

A human readable summary would take that a step further.

Images

Also there is typically an image associated with a news story / article. This may be accessible for use when describing links eg. via a social media "card". We might want to display an image where we show citations.

Related ticked on images: #6586

[garethrees]

Created two tickets:

• Add human-readable citation title #6846
• Add human-readable citation summary #6847

[garethrees]

Linking to the announcement blog post because I always end up here when looking for it https://www.mysociety.org/2020/01/14/has-your-foi-request-been-used-in-a-news-story-now-you-can-let-everyone-know/