Add function to upload files via admin interface

[RichardTaylor]

On WhatDoTheyKnow administrators from time to time upload files under

http://www.whatdotheyknow.com/files/
eg.
https://www.whatdotheyknow.com/request/details_of_meeting_on_the_regula#comment-1648

This is not currently done via the admin interface; it would be useful if it was so that this admin task could be carried out easily by all those with access to the admin interface.

Sometimes files are available on another web server and we want to copy them across to WhatDoTheyKnow (eg. a user has received material by post, scanned it, and temporarily placed it online somewhere for us to obtain and host).

Ideally an "upload" and "obtain file from URL" option would be available.

On rare occasions very large files are involved and the uploading will need to be done by someone with access to the server directly transferring the file rather than going via the web-interface; usually though the web-interface would suffice.

Ideally this feature would link the files to the relevant request once they've been uploaded (perhaps just prompting the uploader to add an annotation containing the file URL(s)?).

[hsenag]

Bringing this within the site makes a lot of sense (depending on priorities/resources of course). The current ssh route is fine for me personally, but it's extra overhead to maintain the credentials and it's another tool to get familiar with if you aren't already.

One other possible requirement: It's frequently an entire directory tree that needs to be uploaded. I'm not sure how easy this is to do from a browser. At present I generally use rsync to do this which means I can interrupt the transfer part-way and restart it without losing my previous progress. Uploading large amounts of data tends to make a home ADSL connection unpleasant to use so being able to interrupt and restart is quite useful.

In an ideal world we would also integrate this with common filesharing services like Dropbox/Google Drive etc but the volume of requests for this is so small that I doubt it's worthwhile.

[RichardTaylor]

At the latest WhatDoTheyKnow call I think it was suggested this feature would be desirable so there was no need to recommend administrators of other Alaveteli installations get involved in directly uploading files to a server.

WhatDoTheyKnow administrators have used Google Drive, with links to the documents from annotations, on a couple of occasions now.

[garethrees]

I created an issue in our sysadmin repo to make files.whatdotheyknow.com an Amazon S3 bucket (https://github.com/mysociety/sysadmin/issues/846) – similar idea to Google Drive, but we can control the domain and access credentials. There should be plenty of GUI S3 clients, so another option here.

[RichardTaylor]

Linking to the related #1243 "Provide or integrate with webcitation-style archiving of externally hosted documents and pages"

[garethrees]

Might be easier and more maintainable with Active Storage once we get to Rails 5.2 (#5069).

[RichardTaylor]

A WhatDoTheyKnow volunteer has told us that if we had this feature they would proactively identify requests where a response has been provided via a link to file-transfer service and upload them to WhatDoTheyKnow.

Currently documents are only uploaded when we are prompted to do so.

The number of file upload requests made to developers might not reflect the number of file uploads which would occur if administrators had direct access to this feature.

[RichardTaylor]

Noting that the bug at

#5905

is resulting in lots of requests to developers to upload files.

[garethrees]

Might be easier and more maintainable with Active Storage once we get to Rails 5.2 (#5069).

We are now on Rails 5.2 so have Active Storage available to us. Will need to consider how to migrate from the existing files dir (preserving URLs) and also how we manage storage options across various installs of Alaveteli.

[garethrees]

Requires #6530

[garethrees]

After https://github.com/mysociety/alaveteli/issues/6532…

class FoiAttachment
  has_one_attached :body
  has_many_attached :replacements

  def body
    replacements.last || super
  end
end

This wouldn't allow uploading of edge-case attachments where e.g. they've provided an egress URL or similar.

Perhaps we want more like:

# Generic upload with a URL
class ManualUpload
  has_one_attached :file
end

class FoiAttachment
  has_one_attached :body
  has_many :replacements

  def body
    replacements.last.file || super
  end
end

# Add a generic manual upload, then link it as a replacement
@foi_attachment.replacements << @manual_upload

[garethrees]

Need to better shape this. Some notes to self:

Issue 1:

• ActiveStorage backend
• Move existing files to new backend
• Preserve the old manually uploaded URLs (Preserve manually uploaded file URLs #6533)

Issue 2:

• Admin UI; can be added later if we're short on time

Issue 3:

• Add ability to "replace" an attachment with a manual upload?

[RichardTaylor]

Add ability to "replace" an attachment with a manual upload?

I like this idea, which would presumably allow a replacement file to be linked in the right place in a thread.

Recognising download, edit, upload as a redaction mechanism might help in terms of presentation, and record keeping for a takedown log / transparency report (#2658)

Consider sometimes a response message is hidden from public view, but we still want to republish an attachment, or attachments, from the message. We might do this where there are problems with publishing some annotations, but not others.

This issue is being considered prior to the feature request for hiding individual attachments (#1005) having been implemented.

[garethrees]

One gotcha with using Active Storage compared to our current approach is that we'd lose the ability to upload and browse directories. Active Storage is just a blob storage mechanism for individual files.

There are two situations where this would be annoying:

1. We upload multiple files because we're preserving what was sent originally in a now hidden message. In the annotation we include the link to the directory rather than each individual file to save time and make the annotations less confusing.
2. We extract a zip file or similar, which itself may contain a full directory tree.

To resolve 1 , if we added files per request (which is what we do currently by placing them in a files/request/:url_title dir) then we could generate a HTML listing of all files attached to the request (/request/:url_title/uploaded_files) and use that URL in the annotation. Alternatively, we could generate the file listing and attach that itself as an Active Storage file, though IDK whether that would then get served correctly.

Not sure how we'd resolve 2 though.

[RichardTaylor]

A work-around here is:

• Create an account with the authority email address
• Confirm the email address manually on the admin page
• Use "login as" to switch to that account
• Use the "respond to request" feature

This might be acceptable if the message content and an associated annotation from the official site account made clear what is happening.

[RichardTaylor]

Variation on the workaround:

• Change the installation's official user account's email address to one at the authority's domain.
• Use "login as" to switch to the official account
• Use the "respond to request" feature to upload files
• Change the email address on the official user account back to what it was
• Add an annotation to explain what has been done

Risk: If anyone used the user-user messaging system to contact the site operators via the official account during the brief period the address was changed such a message would be lost/misdirected. Also if any alert/tracks were generated in that period they would be lost/misdirected too.

Ticket related to the workround, on the ability to upload multiple files at once via the "respond to this request" feature #66

[RichardTaylor]

Following discussion the WhatDoTheyKnow workaround has been developed to use a separate dedicated account to avoid the issues associates with using the official WhatDoTheyKnow account.

The workaround has been documented at:
https://wdtkwiki.mysociety.org/wiki/Uploading_files_to_the_site

[HelenWDTK]

+1 Building this into the system would also make it easier to track existing uploads of frequently requested content (eg train announcements).

[HelenWDTK]

+1 Dealing with these en-masse when they've been batched by the devs adds a considerable admin burden in terms of needing to reacquaint yourself with why the upload was needed in the first place to manually add the note/annotation. It would be much better for admins and users alike if we could add these ourselves as we go along.

Ideally this would display the uploads in a way that makes it clear that we were the ones uploading them. Perhaps using one of the new styled note types that are coming.

[garethrees]

…needing to reacquaint yourself with why the upload was needed in the first place to manually add the note/annotation…

Don't disagree with this being an issue, but could make it a habit to draft the annotation at the point of requesting the file upload, so that it's just a case of copy & pasting it and adding the upload urls when the upload request is fulfilled.

[HelenWDTK]

could make it a habit to draft the annotation at the point of requesting the file upload, so that it's just a case of copy & pasting it and adding the upload urls when the upload request is fulfilled.

Can't do that unfortunately - the timeframe for uploads is such that the circs have often changed.