Upload photo information

[bsaiken]

@damirius
You did not aNswer my question from a different Issue, but this would likely be a new Issue anyway.

Is it possible, when we are uploading a photo, to upload the photo information: timestamp and the file name (as it appears on the mobile device)?

[damirius]

@bsaiken

Sorry about that. Yes it is possible, there's event that is being triggered when we start uploading each image, we already use it for some other stuff. We can make another ajax call there to store that info. What's the purpose of that and where we should store that stuff?

[bsaiken]

@damirius
First of all, we should be storing the timestamp anyway. I will make a new Issue for that.

But, more importantly, this is my big idea! Let's store the file name first. Then, if we get a timeout when the FT is in the field, it should be possible for the FT to initiate an independent upload of the all the photos once they have a WiFi connection. This upload sequence would not be subject to any timeout. We just match the photo to the stored file name.

Tell me if I am correct.

And one more question. Isn't there a comment in dropzone to upload thumbnails?

[damirius]

@bsaiken

Thumbnails are shown in the dropzone field, they aren't actually uploaded on the server. Resize option is popular one dropzone/dropzone#781, but they still didn't add it in the dropzone.
As far as I can see most of the people tried some other solutions or some little hacks to change the code of the dropzone so that it uploads resized images.

[FWAJL]

@damirius @bsaiken

Read http://www.dropzonejs.com/#enqueuing-file-uploads to queue file upload until the ft clicks the start upload:

If you set autoProcessQueue to false, then .processQueue() is never called implicitly. This means that you have to call it yourself when you want to upload all files currently queued.

See if we can retrieve the queue in JSON format, save it to the db and reuse it later for upload

Here are the steps we think we need to take to validate this possible solution:
1- set autoProcessQueue to false on DropZone
2- upload a few files anywhere.
3- retrieve the queue as JSON if possible.
4- save it in cookie or local storage and try to launch a new dropzone "instance" to upload the saved queue.

If those 4 steps are successful, it means we can store the JSON to the database and then later, we retrieve the JSON and repeat step 4 in the situation we want.

Can you try that please @damirius ?

[bsaiken]

@damirius
I don't know if this is helpful or not, but in case...

I dug into dropzone.js. I put a break at line 498 and a watch on the variable file. See attached photo. It seems like that's where the information we need is.

[damirius]

@bsaiken @FWAJL

As I said dropzone is firing few public accessible events on file drop and on file send which can provide us with file variable.
We can queue files and try solution that Jeremie suggested. The question is where will we add that separate window for uploading queued files when FT gets his WiFi connection? Do we need separate window at all or we just need to add button in the infowindow or we will add new page just for that.

[bsaiken]

@damirius
You seem confident this will work. Jeremie and I not so much, so we are taking this one step at a time.

The "Upload Photos" button could go a number of places, and I haven't decided for certain yet. It seemed moot until we knew whether or not it worked. One option is to include a "Wifi Photo Upload" item on the main menu. Another option is to automatically add a checklist item as soon as a queue is initiated. For now, it really does not matter - you could make a dummy page - whatever is easiest for this development stage.

[damirius]

@bsaiken @FWAJL

Yap, that thing with queue doesn't work. Those file objects are used just for the flow control of the process.

[FWAJL]

@damirius @bsaiken so we can't save the files in queue and re-inject them later?

[damirius]

@FWAJL @bsaiken

We can save those "files" objects but they are there just for the control of the upload process, they contain some stuff but we can't inject them later to try and upload the actual files.

[FWAJL]

@damirius @bsaiken Yes, I'm not seeing either the actual info that would link the object file to the physical file itself after. At least for now.

[FWAJL]

@damirius @bsaiken

More on the subject:

Since it is not possible to retrieve the data from Dropzone, I'm investigating a possible solution to store the documents (pictures and pdf) on the FT's device. I'm not sure where yet but I have an idea. Give me a little bit of time to build the case.

The goal would be to use Dropzone to store the data on the FT's device and then to the server when the connection is WIFI.

Thank you for your help so far 👍

[FWAJL]

@bsaiken @damirius

So, I have found a possible solution to test out on a dummy to start with. I found a clear enough example (http://blog.jankuca.com/post/7169506266/offline-file-uploads-file-api) with the source code (https://github.com/jankuca/offline-uploader).
Alternatively, we could use this example: https://github.com/agektmr/BrowserStorageAbuser

Basically, we would use IndexedDb (first choice if available) and WebSQL storage to store the documents on the device. There are quota to respect depending on the platform (iOS or Android if we limit ourselves to those). To find out the limit: go here on your mobile devices here and try to fill the storage with 10 chunks of 5mb (50mb) which I think is a large amount of data.
On my Samsung S4 mini (Android 4.4.2), I can store 50mb in IndexedDB storage and WebSQL.
_I tried add the chunk offline and online and it works._ The only thing is and it may be my phone, the chunk creation is resource heavy

To optimise the storage, I think we should resize images down since recent smartphone can take pictures of several MB. @bsaiken : what would be an acceptable image size in pixels?

I don't know if we can use Dropzone or not. To be confirmed but I think we must try this as it could be the first step to make the whole app offline.

What do you think?

Some more reading and resources:
http://www.html5rocks.com/en/tutorials/offline/quota-research/

[bsaiken]

@FWAJL
That's not enough data. I will have to look for another solution.

[FWAJL]

@bsaiken @damirius @gsouvik

Please definieWhat are your requirements:

• how much do you think you need to store?
• have you tried this link http://demo.agektmr.com/storage/ on your iPad and iPhone? Play with it using chunks of 500kb for example (a picture size big enough to record field parameters for example).
• can you answer this question: what would be an acceptable image size in pixels?

I have tried 100 chunks of 500kb (~50MB) with IndexedDB (max. on android 4.4.2) and it still works, online and offline.
Then I have tried 200 chunks of 500kb (~100MB) with WebSQL and it worked up to 74.8MB because my phone limited space available (done online and offline).

That represents between 100 and 200 documents in total!

We would need to test on lower Android versions (I can test 4.1.2 and 4.4.4) and of iOS. Then we could set a min. platform version that we support because like Souvik said, support is limited but platform updates happens often on recent smartphones.

If you dismiss this solution from my first comment, I only see the native apps as the real solution. These HTML5 technologies are the best chances to stick to the Web App.

Thank you.

[bsaiken]

@FWAJL
I am not in control of the data size requirements. Photos on my phone are ~ 2Mb. Some phones have larger sized images maybe 8Mb or more. I can't really predict how many photos a FT will need to upload. I was thinking 50Mb might be too small. However, if we can limit the file size to 1Mb or less, and we had 50Mb to work with, that would probably be OK. Can you mock this up for me? The only way to really test this is on my mobile phone in a real situation.

[FWAJL]

@bsaiken

I understand. The reason why I'm talking about resizing pictures is because we don't need to full HD pictures to record a field parameter.

However, can you at least provide some start-of-answer to the questions I posed, please:

• how much do you think we need to store based on your experience of field work?
• have you tried this link http://demo.agektmr.com/storage/ on your iPad and iPhone? Play with it using chunks of 500kb for example (a picture size big enough to record field parameters for example).
• can you answer this question: what would be an acceptable image size in pixels?

These questions are to be used to reflect on the need.
I'm not asking how many pictures of FT will upload. We will see first how many _we_ can store to decide if this lead can be implement as a viable solution.

I will try to make you a mockup but it will probably be outside the repository context.

Thank you.

[bsaiken]

@gsouvik @WebDevJL
Jeremie, You may want to merge the DEV branch to feature_devsupport because I uploaded the fine_uploader archive to the DEV branch.

Souvik, This has been a real challenge for us to implement. Let me know if you need help understanding the requirement. We could have a Skype chat if needed. The first step would be to implement a simple upload (1 photo only) from the mobile site. I will create a new Milestone with Issues for each step.

[FWAJL]

@bsaiken @gsouvik merge complete.

[rnicholus]

I'm the maintainer of Fine Uploader. Can I be of assistance here in any way as far as answering further questions?

[bsaiken]

@rnicholus
I bought a Fine Uploader license and I have tech to implement the code. Good to know who to ask if we have questions, though. Thanks.

[rnicholus]

Generally speaking, we like "how do I do this" type of questions to be posted on SO under the fine-uploader tag (which I monitor regularly), and feature/bug reports/requests in our issue tracker. But I'd be happy to provide guidance in this thread for your implementation effort.

[bsaiken]

@rnicholus @gsouvik @FWAJL
Sure, if we have a question this post will notify everyone to ask questions under Fine Uploader's git.
Jeremie may have a follow-up question from my original Stack Overflow question (http://stackoverflow.com/questions/32263174/fine-uploader-delayed-upload-queue).

[rnicholus]

If the questions are very specific to your implementation, you can simply ask in this thread, as I am now watching.

[bsaiken]

from #1282.

@gsouvik said:
technicians just queue up the files and keep the page/modal as it is?

No. This would not help. The technician will need to be able to navigate to other pages and do other work while adding to the upload queue.

@gsouvik said:
The fine-uploader version...doesn't use jQuery.

I just downloaded the wrong library. Please just download the correct library.

[rnicholus]

Note that there is no real advantage to pulling in jQuery-wrapped Fine Uploader. See FineUploader/fine-uploader#1310 for more details.

[gsouvik]

@rnicholus
Thanks for sharing that thread on jQuery wrapper, it really helped.
I need a bit of more information regarding a different area of the library. Is it possible to store up the queue itself and upload it at a later time? I'll try to explain that with an example. Say I have a page where I've configured fineuploader with manual upload trigger. I add some files to upload, but don't start the upload. I navigate away to some other page or close the browser. Is it possible for me to get back the entire queue when I open this page again, so that I may start the upload. The idea is, I'm in a situation where I need to create the queue but must be able to upload it at a later time without re-selecting the files.
Thank you.

[rnicholus]

I navigate away to some other page or close the browser. Is it possible for me to get back the entire queue when I open this page again, so that I may start the upload.

No. To support this, one of the following would have to be happen:

1. Each file would have to be stored in the browser. One option is localStorage but space in that store is limited. localStorage is meant for small amounts of text, such as JavaScript objects. Simple key/value store. Another option would be to store the files in IndexedDB. This is possible, but not currently supported by Fine Uploader. Even if we did support this, it would be limited to very new browsers (nothing older than IE10) and managing this type of data in IndexedDB would likely add quite a bit more complexity to Fine Uploader.
2. Another alternative would be to store the paths to the files on the user's local filesystem. In theory, this seems simple, but there is no way to gain unrestricted access to the user's filesystem, for obvious security reasons. The closest thing to this type of access is via a sandboxed filesystem provided by the Filesystem API specification. Currently, that W3C spec is deprecated, and only Chrome implements it.

[bsaiken]

@gsouvik @rnicholus
One of the advantages of not being as tech savvy as the rest of you is that I'm too stupid enough to realize it won't work. I see three possibilities where this could still work - correct me if I am wrong.

1. Specify the exclusive use of Chrome. The only disadvantage is one might expect Filesystem API would eventually be deprecated in Chrome as well.
2. Store the files in IndexedDB. I understand this is not supported by fine-uploader - what level of effort would it take to code a fine-uploader plug-in that would make it work on our site?
3. Since the limitation arises when we close the browser and/or navigate away from the page, how about this:
   First - assume the user would not close the browser until the photos in the queue are uploaded.
   Second, could we create a separate upload page that stays open as a separate tab or separate window? Each time the user selects a new file to upload, they are directed to the "upload" page, where the queue is staged. On the upload page there are buttons to either a) return to the origination page (in the other window or tab) or b) upload the queued photos.

[rnicholus]

Specify the exclusive use of Chrome. The only disadvantage is one might expect Filesystem API would eventually be deprecated in Chrome as well.

We are already making use of the Filesystem API to accept folder uploads via DnD and the <input type="file"> element. But since this is a dead specification, the implementation in Chrome/Opera is now considered proprietary/non-standard. Even though Google will likely keep this in Blink until a suitable replacement arrives in the form of a new standard, it's probably not wise to code against this any further. As such, we will not invest any additional time/resources building upon this implementation.

Store the files in IndexedDB. ...what level of effort would it take to code a fine-uploader plug-in that would make it work on our site?

I'm not sure at the moment, as this is not a web specification that I have much practical experience with. I am fairly certain that this will not be a trivial feature to implement, and I am hesitant to code up a feature as part of the library that can potentially persist large amounts of data to a user's filesystem. This feature could not be unbounded. Complexity follows. However, you could certainly code this up on your end. Simply grab each file from Fine Uploader's API as it is submitted via the getFile method, store it in IndexedDB, and them remove it once the file has been uploaded. When an instance is created, check IndexedDB for files. If any exist, feed them into Fine Uploader via the addFiles API method.

[bsaiken]

@rnicholus @gsouvik
Ray, I am not expecting any code modifications to fine-uploader on your part, my query to you is only to know if it can be done and to verify that any modifications we make for incorporation into our website are not an infringement on code that is rightfully yours. If it does turn out we are able to make modifications, and you like them, you would be welcome to it.

The question of the level of effort required is directed to Souvik, who would take on the work. If the coding is too complex (a.k.a. expensive) we may need to stand down - I'm a one man shop with limited resources and guile can only get me so far.

[rnicholus]

The concept is straightforward, but the devil is always in the details. Though this should be a possible integration given Fine Uploader's comprehensive API.

Your biggest obstacle may be browser support. First, only Firefox & Chrome/Opera fully support IndexedDB. IE10+ and Safari for OSX only have partial support for the spec. Also, in typical Apple form, the implementation of IndexedDB was horribly botched in iOS8. They haven't fixed it in iOS9 either as far as I know.

I honestly can't give you an accurate estimate as to how much work this will be as I am not very familiar with the details of this API as I have not used it before for anything non-trivial. There is an appropriate example that you will likely benefit from at https://hacks.mozilla.org/2012/02/storing-images-and-files-in-indexeddb/.

All said, this doesn't seem like a particularly difficult API to work with (famous last words). Also, I think this will be an interesting integration. If Fine Uploader work ever dies down a bit, I may consider writing a blog post detailing this type of workflow support.

[bsaiken]

@rnicholus wrote:
Your biggest obstacle may be browser support.

This is not an obstacle for me since I can specify a browser to my users (if necessary). This is no more restrictive than requiring users to access the site with a custom mobile app.

@gsouvik - Souvik, when you consider how we might implement this, I believe our best option is to use Chrome. All my testing will be using Chrome. We only need the mobile UI to work on the mobile version of one browser.

[rnicholus]

Keep in mind that Chrome on iOS is, for the most part, just a less
performant version of Safari, with less features. It does not share the
same feature set or codebase as its desktop cousin.
On Wed, Sep 16, 2015 at 12:20 AM bsaiken notifications@github.com wrote:

@rnicholus https://github.com/rnicholus wrote:
Your biggest obstacle may be browser support.

This is not an obstacle for me since I can specify a browser to my users
(if necessary). This is no more restrictive than requiring users to access
the site with a custom mobile app.

@gsouvik https://github.com/gsouvik - Souvik, when you consider how we
might implement this, I believe our best option is to use Chrome. All my
testing will be using Chrome. We only need the mobile UI to work on the
mobile version of one browser.

—
Reply to this email directly or view it on GitHub
#1250 (comment)
.

[gsouvik]

@bsaiken
After going through the discussion thread, from my view point, the following two options seems to be somewhat feasible:

1. As you suggested, using a separate upload page and keep adding everything to it. The user must not close the tab (the page loaded in it) or else they loose the queue. They must be educated regarding this.

2. IndexedDB - I haven't worked with it before, but can research and see how much it might work. I think @WebDevJL already did some research on it and has some data, we may use those, though I may be wrong.

Either way, to store the queue is the biggest challenge and I think we need to do some experiments before we could actually adopt one of the options. Please share your thought, which one you would want me to try.

[WebDevJL]

@bsaiken @gsouvik

Yes, you're right, Souvik. Look at this comment for details about IndexedDb / WebSQL implementation example: #1250 (comment)

[bsaiken]

@WebDevJL
Jeremie, your opinion about the upload page?

[gsouvik]

@rnicholus @bsaiken
I was using the scaling feature of the fineuploader library and facing some issues. Let me explain step by step. Following is my fineuploader javascript setup:

      var galleryUploader = new qq.FineUploader({
        element: document.getElementById('my-uploader'),
        template: 'qq-template-gallery',
        request: {
          endpoint: config.rootFolder + 'file/upload',
          inputName: 'file'
        },
        form: {
          element: document.getElementById('document-upload'),
          autoUpload: true
        },
        thumbnails: {
            placeholders: {
                waitingPath: 'placeholders/waiting-generic.png',
                notAvailablePath: 'placeholders/not_available-generic.png'
            }
        },
        validation: {
            allowedExtensions: ['jpeg', 'jpg', 'png']
        },
        scaling: {
          sendOriginal: false,
          includeExif: false,
          sizes: [
            {name: "small", maxSize: 200}
          ]
        }
      });

I've used scaling as well as attached a form with it so that everything works with my already coded end point. I've changed the file input name from qqfile to file so that I do not have to change my end point.

When I debug the _files superglobal on the server end point this is the output I see:

Array
(
    [file] => Array
        (
            [name] => blob
            [type] => image/jpeg
            [tmp_name] => /private/var/tmp/phpbjAFe0
            [error] => 0
            [size] => 9010
        )

)

As you see, the scaled version of the file actually uploads but the original name of the file (which according to fineuploader documentation should have been: "original name" + "small.extension")
of the file is not maintained and my end point fails.

Could you please let me know if I'm making any configuration mistake or how I may fix it. Let me know if you need more info on the issue.

Thank You.

[rnicholus]

The name of the scaled file is stored in the qqfilename request parameter.

On Wed, Sep 23, 2015 at 12:44 PM Souvik Ghosh notifications@github.com
wrote:

@rnicholus https://github.com/rnicholus @bsaiken
https://github.com/bsaiken
I was using the scaling feature of the fineuploader library and facing
some issues. Let me explain step by step. Following is my fineuploader
javascript setup:

  var galleryUploader = new qq.FineUploader({
    element: document.getElementById('my-uploader'),
    template: 'qq-template-gallery',
    request: {
      endpoint: config.rootFolder + 'file/upload',
      inputName: 'file'
    },
    form: {
      element: document.getElementById('document-upload'),
      autoUpload: true
    },
    thumbnails: {
        placeholders: {
            waitingPath: 'placeholders/waiting-generic.png',
            notAvailablePath: 'placeholders/not_available-generic.png'
        }
    },
    validation: {
        allowedExtensions: ['jpeg', 'jpg', 'png']
    },
    scaling: {
      sendOriginal: false,
      includeExif: false,
      sizes: [
        {name: "small", maxSize: 200}
      ]
    }
  });

I've used scaling as well as attached a form with it so that everything
works with my already coded end point. I've changed the file input name
from qqfile to file so that I do not have to change my end point.

When I debug the _files superglobal on the server end point this is the
output I see:

Array( [file] => Array ( [name] => blob [type] => image/jpeg [tmp_name] => /private/var/tmp/phpbjAFe0 [error] => 0 [size] => 9010 ))

As you see, the scaled version of the file actually uploads but the
original name of the file (which according to fineuploader documentation
should have been: "original name" + "small.extension")
of the file is not maintained and my end point fails.

Could you please let me know if I'm making any configuration mistake or
how I may fix it. Let me know if you need more info on the issue.

Thank You.

—
Reply to this email directly or view it on GitHub
#1250 (comment)
.

[gsouvik]

@rnicholus
Is there any Javascript configuration to override it so that it uses file when the original file is not uploaded?

[rnicholus]

No, and in fact this is not possible cross-browser.