feature request: upload binary files into the repo

[l3iggs]

It would be quite handy for me if, when I click the 'New' button via the web interface, I would be presented with the option to either add a new page or add a binary file directly into the wiki's repository (via upload). This would be great for adding images into the wiki while ensuring the whole thing stays nice and self-contained and portable.

[dekimsey]

I've been thinking of something like this. Personally I'd go a step further and provide a "binary-editor" interface for unrecognized mime-types that'd provide an interface to upload new binaries. I haven't put forth much effort into thinking about how this would need to be engineered to avoid security vulnerabilities. But it has been an idea I've been tossing around.

[l3iggs]

Yeah, a special web-based editor for various types of files would be pretty great for sure, it just seems like feature creep could become a problem here. Some relatively simple upload code would really go a long way to supporting a stand-alone wiki use case.

As an interm solution, one could always download whatever file they wish to edit from the wiki, edit it locally and then just update it by re-uploading (if it were to become possible to upload a file through the web interface). :^)

[bootstraponline]

We've mostly tried to keep the wiki similar to what's available on GitHub.com. File upload is one of the use cases that isn't covered well. I agree it'd be a useful feature though.

[l3iggs]

I've managed to hack something together that seems to work for me here. It allows me to upload files through the web interface into a folder called uploads that gets committed and tracked with the wiki. Here's how I did it in case anyone is interested.

I added an upload button to the page view:

diff --git a/lib/gollum/templates/page.mustache b/lib/gollum/templates/page.mustache
index edffb49..01a6795 100644
--- a/lib/gollum/templates/page.mustache
+++ b/lib/gollum/templates/page.mustache
@@ -22,6 +22,8 @@ Mousetrap.bind(['e'], function( e ) {
       <a href="#" id="minibutton-new-page">New</a></li>
     {{#editable}}
     <li class="minibutton jaws">
+      <a href="#" id="minibutton-upload-page">Upload</a></li>
+    <li class="minibutton jaws">
       <a href="#" id="minibutton-rename-page">Rename</a></li>
     <li class="minibutton"><a href="{{base_url}}/edit/{{escaped_url_path}}"
        class="action-edit-page">Edit</a></li>

then made it do something:

diff --git a/lib/gollum/public/gollum/javascript/gollum.js b/lib/gollum/public/gollum/javascript/gollum.js
index afbc0f2..2fafade 100755
--- a/lib/gollum/public/gollum/javascript/gollum.js
+++ b/lib/gollum/public/gollum/javascript/gollum.js
@@ -144,6 +144,25 @@ $(document).ready(function() {
       $('body').addClass('ie8');
     }
   }
+  
+  if ($('#minibutton-upload-page').length) {
+    $('#minibutton-upload-page').parent().removeClass('jaws');
+    $('#minibutton-upload-page').click(function(e) {
+      e.preventDefault();
+
+      $.GollumDialog.init({
+        title: 'Upload File',
+        fields: [
+          {
+            type:   'upload'
+          }
+        ],
+        OK: function( res ) {
+               
+        }
+      });
+    });
+  }

   if ($('#minibutton-rename-page').length) {
     $('#minibutton-rename-page').parent().removeClass('jaws');

i added an upload case to your dialog box (couldn't manage to tie this form's commit to the existing OK button though):

diff --git a/lib/gollum/public/gollum/javascript/gollum.dialog.js b/lib/gollum/public/gollum/javascript/gollum.dialog.js
index 969842a..c1ef056 100755
--- a/lib/gollum/public/gollum/javascript/gollum.dialog.js
+++ b/lib/gollum/public/gollum/javascript/gollum.dialog.js
@@ -41,6 +41,13 @@
           case 'text':
             fieldMarkup += Dialog.createFieldText( fieldArray[i] );
             break;
+          
+          case 'upload':
+               fieldMarkup += '<form action="uploadFile" method="POST" enctype="multipart/form-data">' +
+               '<input type="file" name="file" id="file"><br>' +
+               '<input type="submit" name="submit" value="Upload">' +
+               '</form>';
+               break;

           default:
             break;

And here's the code that moves the file into the repo after it's uploaded and commits it. I threw a git push in there because I like to keep my wiki backed up remotely. I tried to use your committer here but couldn't get that to work for some reason.

diff --git a/lib/gollum/app.rb b/lib/gollum/app.rb
index 1a043a5..07a3bda 100644
--- a/lib/gollum/app.rb
+++ b/lib/gollum/app.rb
@@ -5,6 +5,7 @@ require 'gollum-lib'
 require 'mustache/sinatra'
 require 'useragent'
 require 'stringex'
+require 'fileutils'

 require 'gollum'
 require 'gollum/views/layout'
@@ -13,6 +14,10 @@ require 'gollum/views/has_page'

 require File.expand_path '../helpers', __FILE__

+#required to upload bigger binary files
+Grit::Git.git_timeout = 120 # timeout in secs
+Grit::Git.git_max_size = 190 * 10**6 # size in bytes (10^6=1 MB)
+
 # Fix to_url
 class String
   alias :upstream_to_url :to_url
@@ -88,6 +93,13 @@ module Precious
       @css = settings.wiki_options[:css]
       @js = settings.wiki_options[:js]
     end
+    
+    after do
+      wikip     = wiki_page('home')
+      wiki      = wikip.wiki
+      wiki.repo.git.push
+      
+    end

     get '/' do
       page_dir = settings.wiki_options[:page_file_dir].to_s
@@ -145,6 +157,30 @@ module Precious
         redirect to("/create/#{encodeURIComponent(@name)}")
       end
     end
+    
+    post '/uploadFile' do
+      if params[:file]
+        filename = params[:file][:filename]
+        tempfile = params[:file][:tempfile]
+       uploadDir = "uploads"
+        #uploadDir = ::File.join(settings.gollum_path,"uploads")
+       STDOUT.write uploadDir
+        fileTarget = ::File.join(uploadDir,filename)
+        if !FileTest::directory?(uploadDir)
+          Dir::mkdir(uploadDir)
+        end
+        
+        FileUtils.mv(tempfile.path,fileTarget)
+
+      end
+      wikip     = wiki_page('home')
+      wiki      = wikip.wiki
+      
+      wiki.repo.add(fileTarget)
+      wiki.repo.commit_index("Uploading binary file to uploads/#{filename}")
+
+      redirect to("/home")
+    end

     post '/rename/*' do
       wikip     = wiki_page(params[:splat].first)

[JangoSteve]

@l3iggs I applied your patch and it worked great. It could use a little cleaning up (like perhaps tying the upload action to the "OK" button of the dialog instead of a separate "Upload" button, and also adding buttons to the individual page actions that allow you to upload the image and insert it into a page all at once), but it worked like a charm. I couldn't find your code in a fork or anything, so I just created a pull request from it myself. Please let me know if you were going to create one and I can withdraw mine.

[l3iggs]

Awesome! Thank you very much for putting this together. I'm really glad it works for you. The implementation is pretty hacky for sure, but hopefully somebody out there can take what's here and clean it up better than I can.

Have you tried uploading large files, >10MB? I seem to remember there was a limitation there that I've since solved.

[JangoSteve]

Hmm, no I haven't tried that yet, I've just been using it to upload images and screenshots.

[l3iggs]

Yeah, image upload what I typically use it for too. I think I tried to upload a large tiff file and ran into the issue. So if your image files are smaller than some limit (can't remember off hand) then the current code should give you no issues.

[amenonsen]

It's easy enough to tie the file upload to the dialog's OK button. Here's how I did it.

diff --git a/lib/gollum/public/gollum/javascript/gollum.dialog.js b/lib/gollum/public/gollum/javascript/gollum.dialog.js
index 969842a..82407c1 100755
--- a/lib/gollum/public/gollum/javascript/gollum.dialog.js
+++ b/lib/gollum/public/gollum/javascript/gollum.dialog.js
@@ -37,11 +37,14 @@
           fieldMarkup += '<div class="field">';
           switch ( fieldArray[i].type ) {

-          // only text is supported for now
           case 'text':
             fieldMarkup += Dialog.createFieldText( fieldArray[i] );
             break;

+          case 'file':
+            fieldMarkup += Dialog.createFieldFile( fieldArray[i] );
+            break;
+
           default:
             break;

@@ -86,6 +89,22 @@
       return html;
     },

+    createFieldFile: function( fieldAttributes ) {
+      // Not actually a field, but an embedded form.
+      var html = '';
+
+      var id = fieldAttributes.id || 'upload';
+      var name = fieldAttributes.name || 'file';
+      var action = fieldAttributes.action || 'uploadFile';
+
+      html += '<form method=post enctype="multipart/form-data" ' +
+        'action="' + action + '" ' + 'id="' + id + '">';
+      html += '<input type=file name="' + name + '">';
+      html += '</form>';
+
+      return html;
+    },
+
     createMarkup: function( title, body ) {
       Dialog.markupCreated = true;
       if ($.facebox) {

And then:

diff --git a/lib/gollum/public/gollum/javascript/gollum.js b/lib/gollum/public/gollum/javascript/gollum.js
index afbc0f2..5e5bf85 100755
--- a/lib/gollum/public/gollum/javascript/gollum.js
+++ b/lib/gollum/public/gollum/javascript/gollum.js
@@ -145,6 +145,25 @@ $(document).ready(function() {
     }
   }

+  if ($('#minibutton-upload-page').length) {
+    $('#minibutton-upload-page').parent().removeClass('jaws');
+    $('#minibutton-upload-page').click(function(e) {
+      e.preventDefault();
+
+      $.GollumDialog.init({
+        title: 'Upload File',
+        fields: [
+          {
+            type:   'file'
+          }
+        ],
+        OK: function( res ) {
+          $('#upload').submit();
+        }
+      });
+    });
+  }
+
   if ($('#minibutton-rename-page').length) {
     $('#minibutton-rename-page').parent().removeClass('jaws');
     $('#minibutton-rename-page').click(function(e) {

I did some cleanups in gollum.dialog.js that aren't technically necessary. All I really needed to do was stick an 'id="upload"' onto the generated upload form, so that I can select and submit it in the OK callback.

I'm not especially fond of conflating a field with a form as this code does, but I'm not sure all the changes needed to make it look cleaner are really justified. What I have above works well enough.

Thanks to @l3iggs for the original patch, much appreciated.

[amenonsen]

As for "I tried to use your committer here but couldn't get that to work for some reason", that was a lot harder to fix, but here's an attempt to do so.

I chose to use Gollum::Committer, since that has the necessary infrastructure to specify author information and update the working directory after a change. (I could have used Grit's index directly, but then I would have had to reinvent both the duplicate-checking in add_to_index and the update_working_dir method.)

One problem with this is that I have to invent a "format" for the files. If the file has an extension, I retain it; for files without extensions, I arbitrarily chose to use '.txt'. (Unfortunately, files with unknown extensions don't show up in Gollum at all, though they're still accessible by their full path. But that's something I haven't looked into. If anyone has suggestions, I'm all ears.)

Here's my diff:

diff --git a/lib/gollum/app.rb b/lib/gollum/app.rb
index 516ba9e..c4f0269 100644
--- a/lib/gollum/app.rb
+++ b/lib/gollum/app.rb
@@ -13,6 +13,10 @@ require 'gollum/views/has_page'

 require File.expand_path '../helpers', __FILE__

+#required to upload bigger binary files
+Grit::Git.git_timeout = 120 # timeout in secs
+Grit::Git.git_max_size = 190 * 10**6 # size in bytes (10^6=1 MB)
+
 # Fix to_url
 class String
   alias :upstream_to_url :to_url
@@ -146,6 +150,46 @@ module Precious
       end
     end

+    post '/uploadFile' do
+      if params[:file]
+        fullname = params[:file][:filename]
+        tempfile = params[:file][:tempfile]
+      end
+
+      dir = 'uploads'
+      ext = ::File.extname(fullname)
+      format = ext.split('.').last || 'txt'
+      filename = ::File.basename(fullname, ext)
+      contents = ::File.read(tempfile)
+      reponame = filename + '.' + format
+
+      wiki = wiki_new
+      head = wiki.repo.head
+
+      options = {
+        :message => "Uploaded file to uploads/#{reponame}",
+        :parent => wiki.repo.head.commit,
+      }
+      author = session['gollum.author']
+      unless author.nil?
+        options.merge! author
+      end
+
+      begin
+        committer = Gollum::Committer.new(wiki, options)
+        committer.add_to_index(dir, filename, format, contents)
+        committer.after_commit do |committer, sha|
+          wiki.clear_cache
+          committer.update_working_dir(dir, filename, format)
+        end
+        committer.commit
+        redirect to("/home")
+      rescue Gollum::DuplicatePageError => e
+        @message = "Duplicate page: #{e.message}"
+        mustache :error
+      end
+    end
+
     post '/rename/*' do
       wikip     = wiki_page(params[:splat].first)
       halt 500 if wikip.nil?

(I've never written any Ruby code before, so please excuse any stupid errors in the above. I welcome any comments about how I could have done this better.)

[amenonsen]

I wrote: "(Unfortunately, files with unknown extensions don't show up in Gollum at all, though they're still accessible by their full path. But that's something I haven't looked into. If anyone has suggestions, I'm all ears.)"

For the record, I found the answer browsing older tickets: run gollum with --show-all