AWS Bucket Alternative

[denjello]

Our NGO is considering adopting loomio, but we have a strict policy preventing me from using off-site storage. Furthermore, we do not have the facilities to spin up 3 riak-cs servers to "fake the buckets".

I understand the methodology behind using federated services like facebook and google authentication as options for logging in and registering. These are nice to have, but there is still the option to register "normally". It would be grand to be able to use a similar "graceful degradation" back to the native filesystem if for whatever reason AWS is not appropriate.

Is there a fork or an approach that could be recommended in this case?

[robguthrie]

Yep. Totally understand and agree. We support direct uploads for profile images, but not attachments.

If you have someone willing to do the work, or are willing to sponsor someone to do the work, then it would not be too much effort to support direct uploading for attachments using the same systems that profile images use,

[denjello]

Thanks for the heads up on where to look. :) Once I get it working I'll submit a pull request.

[robguthrie]

Great to hear you're able to help!

May I recommend that you use an environment variable to switch between current behaviour and new behaviour in the UI?

Another tip: the Attachment model could be extended to support a Paperclip attachment, and you could add a method that returns the attachment url from either system.

[robguthrie]

Please feel free to ask me for any help in figuring this out, good luck!

[denjello]

I had planned on using an environment variable to do that. 👍

[denjello]

for the record here is the REALLY quick fix:

/config/application.rb

+   config.paperclip_defaults = {
+     :storage => :filesystem
+   }
-    config.paperclip_defaults = {
-      :storage => :fog,
-      :fog_credentials => {
-        :provider => 'AWS',
-        :aws_access_key_id => Rails.application.secrets.aws_access_key_id,
-        :aws_secret_access_key => Rails.application.secrets.aws_secret_access_key
-      },
-      :fog_directory => Rails.application.secrets.aws_bucket,
-      :fog_public => true
-    }

[denjello]

So here is the solution. Simply changing the paperclip_defaults will set everything to be uploaded to “:rails_root/public/:attachment/:id/:style/:basename.:extension” as per the docs: http://www.rubydoc.info/gems/paperclip/Paperclip/Storage/Filesystem
Here is a patch because I am not using a github fork at the moment.

You can fix up your ENV with the following shell command (assuming you built Loomio according to the developer installation guide):

If you want to save files to the filesystem:

$ echo 'UPLOAD_TYPE="filesystem"' >> ~/projects/loomio/.example-env

If you want to save files to AWS:

$ echo 'UPLOAD_TYPE="AWS"' >> ~/projects/loomio/.example-env

Disclaimer: I am not sure what will happen if you already have files logged in your database and then switch from filesystem to AWS.

[denjello]

I guess I jumped the gun on that one. The solution above is only valid pre 1.0 and only works for avatars, group banners and group icons. IT IS NOT A SOLUTION FOR COMMENT IMAGES.

[robguthrie]

I suspect that it's because our attachment uploading is really custom
compared to the profile photo uploading.

What is blueimp?

On Mon, Jan 26, 2015 at 5:22 AM, denjell notifications@github.com wrote:

I guess I jumped the gun on that one. I can upload avatars, logos etc. but
not attachments for some reason. I suspect it is blueimp :/

—
Reply to this email directly or view it on GitHub
#1846 (comment).

[denjello]

Yeah @robguthrie - and I think that that is a total bummer because it wouldn't have to be that way if the whole world wide web had always ran on concepts like we see in angular and websockets. I guess I am at a bit of a loss as to where I should stick my time as a developer here.

My feeling is that the smartest thing to do is to go the entire upload thing the angular way. No more code-bloat through individual systems for profiles, banners, comments, images, videos, docs and zips but a singular API structure to handle single-page uploading with server-side mimetype verification and virus detection, as well as a scaffolding that allows for immediate presentation according to delivered file type and finally a storage system that puts files in that place that the administrator has set up - be it FS, NFS, ZFS, RIAK-CS, FAKE-AWS, REAL-AWS or whatever.

This is my reasoning: The paperclip gem uses the file binary for mime-type reporting - which is good because it is pretty cross-platform but weak because it is only able to identify 90% of extant filetypes. (I tend to use mdg-mime.) For most uses this is good enough, but unfortunately, my understanding of the current file upload system is such that the only time the mime-type is checked during comment uploads is when the client's browser reports it as e.g. image/png. In which case it is okay for the constructed AWS query and will be delivered as uploaded because the AWS post request has been prepared such that it is valid before the user has even decided to upload something. Ouch.

[edit]
http://threatpost.com/png-image-metadata-leading-to-iframe-injections/104047
http://de.slideshare.net/saumilshah/deadly-pixels-nsc-2013
http://blog.sucuri.net/2014/02/new-iframe-injections-leverage-png-image-metadata.html
[/edit]

By not sanitizing user uploaded data, even in the case of mime-valid images, users are exposed to unsanitised files and this is a risk I am not willing to take on the behalf of the users of the system I administer - because some layers will remain public. For my "beta" deploy, I will probably just turn off the comment upload feature and instruct my users to apply markdown formatting to embed images that they have uploaded somewhere else. I know, it only skirts the problem, but I don't really see any other way around this roadblock other than what I wrote in the second paragraph.

p.s. The blueimp lib is referenced in /app/views/attachments/iframe_upload_result.html.haml and called in /app/views/attachments/new.html.haml ---- But blueimp wasn't really the problem - in and of itself - just a symptom of a malady that is begging for a unified solution.

[gdpelican]

The new attachments system will allow local file uploads, or using other providers with a little environment tweaking. I'll be providing some docs around this in the upcoming days, which will make it into the tech manual as well.