Safari iOS, images don't display if over the resource limit (and useCanvas = true)

[monospaced]

Preamble:

Images in Safari iOS are subject to resources limits

In essence, newer iOS devices have an image area limit of 5,242,880 (5 * 1024 * 1024)

Step to recreate issue:

• Make an image (PNG, JPG, etc) that is over the iOS resource limit, e.g. 2984px x 1757px = 5,242,888px

• open this image in OpenSeadragon via a Legacy Image Pyramid - the image will not load:

{
  "tileSources": {
    "type": "legacy-image-pyramid",
    "levels": [{
      "url": "images/5242888.jpg",
      "width":  2984,
      "height": 1757
    }]
  }
}

• change useCanvas to false - the image will now load.

---

I'm not sure if it's the canvas limit or the image limit that is causing this issue. But the threshold image area required for it not to display is exactly 5,242,880.

If the same over-limit image is opened directly in the browser, it will be displayed, because the browser automatically reduces the size to bring it under the limit.

Perhaps this is not happening in OpenSeadragon because the image is being opened programatically by JS so it can be drawn to the canvas?

It is possible to work around this issue by setting useCanvas to false, perhaps by including an additional User Agent sniff in the $.supportsCanvas function. But this is quite unsatisfactory of course.

I'll continue to look into this issue, but thought I would log it here for the community also :)

[msalsbery]

Good catch!

Not sure what a reasonable fix would look like...

[monospaced]

Unsatisfactory external workaround:

Microfiche.DEFAULTS = {
  useCanvas: navigator.userAgent.match(/(iPad|iPhone|iPod)/g) ? false : true
};

[iangilman]

So that's the limit for a single image tile? Seems like one should use image tiles that are smaller than that anyway.

[msalsbery]

Canvas element is the worst restriction...affects the legacy tile source where there is only one tile.

"JPEG images can be up to 32 megapixels"

"The maximum size for a canvas element is 3 megapixels for devices with less than 256 MB RAM and 5 megapixels for devices with greater or equal than 256 MB RAM."

[iangilman]

Ah, because after we load the tile image, we render it to an offscreen canvas, right?

So it seems the fix would be to detect if you're on iOS and automatically scale down any oversized tiles when rendering them to the offscreen canvas (and of course compensate for that when rendering it to the viewer canvas).

For that matter, why do we render the tiles to offscreen canvases? I assume it makes the viewer canvas rendering more efficient? It also provides us with the opportunity to post-process the tiles before drawing.

[monospaced]

Legacy image pyramid, when you only have one tile, is a very useful feature :)

I've made some further investigations into this issue.

Yep, rendering to the offscreen canvas is where the problem occurs. It's possible to get over-resource-limit tiles to render, just by hacking values for canvas.width and canvas.height (to bring the total area below 5,242,880):

https://github.com/openseadragon/openseadragon/blob/master/src/tile.js#L273

Of course, the tile is then cropped, but in principle @iangilman's solution would work.

I had a go at adding the fix, but ran into a couple of issues:

Firstly, the maths required to find the maximum (proportional) width/height values to bring the total area below the 5,242,880 limit. This one is solvable of course!

Secondly, once you've detected iOS, detected that the image area is over the limit, and found the required dimensions to resize the image to, you then hit a separate issue - that iOS can apply unexpected subsampling and improper stretching to resized images drawn to canvas!

http://stackoverflow.com/questions/11929099/html5-canvas-drawimage-ratio-bug-ios/

Here is a library to resolve this:

https://github.com/stomita/ios-imagefile-megapixel

However, at this point, my quick fix had turned into quite a lot of code, and possibly implementing a third-party library...

Does not seem that reasonable, so I think these are the options to consider:

1. continue on the path of adding this fix, tackling the subsampling issue, tricky maths, etc.
2. perhaps OpenSeadragon doesn't need to render tiles to offscreen canvas as @iangilman has pondered?
3. stick with a 'no canvas for iOS' solution (after all, it seems there are a lot of caveats to iOS canvas support!)

These issues could also impact the viewer canvas if it had a large enough area.

Hope this is helpful info :)

[iangilman]

We definitely want to embrace Canvas going forward... a number of new features we're considering really only work with Canvas. It would be great if we could find a way to support it on iOS. That said, I suppose we could consider a special case only for the legacy image pyramid.

I think there are good reasons to render to offscreen canvases (or at least be able to), though it would be good to verify that it does make a difference from a speed standpoint. At any rate, perhaps we could have a special case for iOS + legacy pyramid where it doesn't do that.

A fourth option might be to load large legacy images and then chop them into Canvas tiles on the client side. This won't hit the iOS resize bug because it's cropping instead of resizing, right? One advantage to this is we wouldn't have to special-case any of the rendering code (though I don't know what complications it would add to the drawing and tile loading code).

[terenceingram]

Thanks @iangilman @monospaced for my solution was I detect the the iOs browser and then have a look to keep to reducing the dimensions of the image until it's under the threshold. I then make the request to the server which dynamically resizes the image. Not great but it least it works.

[iangilman]

Thanks for the update!

[nasjonalmuseet]

I'm using OpenSeadragon for a museum collection and ran into this iOS isssue. I'm pulling the images from an external server and can't and won't copy and resize them on our server. I hoped that reducing the canvas size to below 5 megapixels in the tileSources would solve the issue, but as been noted here before it is reading the actual image file in the browser that causes the problem.
However, I did the maths for reducing the scale whilst keeping proportions and all that and thought I should share it if anybody wants to save themselves some headache.

    // if image bigger than the iOS limit of 5,242,880 px
    if(( $originalWidth * $originalHeight) > 5242880 ) {
        // divide the original size with the limit and get the square root
        $divisor = sqrt( ($originalWidth * $originalHeight) / 5242880 );
        // divide height and width with the divisor to get the new lengths
        // these two multiplied will make exactly 5,242,880
        $newHeight = (int)( $originalHeight / $divisor );
        $newWidth = (int)( $originalWidth / $divisor);
    }

Now I'm hoping that someone has a nice fix : )

[nharn85]

Was there any progress on this bug? If not I am unsure how to make use of the library noted above or the snippet. My issue (just in case) is that the single image won't load on iOS at all. I am not tiling, only 1 image that seems too large for iOS.

Any assistance would be appreciated!

[iangilman]

It doesn't look like there has been any progress (what you see here is what you get).

Is it possible for you to tile your images? If so, that's your easiest fix.

Another possibility might be to use the useCanvas: false option when creating your viewer. This may bypass the issue entirely, though the noncanvas viewer may have other issues.

Otherwise, if you're up for trying to fix the issue in OSD I'm happy to support your effort. The first step would be to figure out which strategy to try of the above mentioned ideas. I'm not sure what's best; maybe try a couple strategies and see which works. I'm thinking:

• See about not storing the tiles in offscreen canvases but instead drawing the image directly to the onscreen canvas as needed. This would be a little bit like what we already do when there's no canvas support at all.
• See about breaking the image into offscreen canvas tiles when it's first loaded and use that for the tile drawing. This would be similar to the pyramid building ImageTileSource does.

You'd want to do something relatively quick and dirty just to verify that one of those strategies solves the problem, and then refine after that.

[terenceingram]

@nmacd85 If you are only loading one image I suggest resizing it. As it's only one image hopefully you can do that statically. My solution was using IIP Server so I was able to resize on the fly. However we moved to using the deep zoom functionality (DZI) and this problem went away. Probably because each individual tile in DZI is well under the problematic threshold.

[nharn85]

@terenceingram thanks for the reply, I was able to get DZI working also! I think I renamed something I shouldn't have originally which caused the issue and me looking for ways to use a single image. Luckily, all set! I haven't officially started development yet but I just need to get the overlays working as expected.

[terenceingram]

@nmacd85 we are very happy using DZI. It's worked a treat on a our historic pictures, digitised books, manuscripts and map collections. I originally was using the legacy image pyramid but once I hooked up DZI we were floored at the result. Here's an example of a panorama: http://nla.gov.au/nla.obj-151651939 and map: http://nla.gov.au/nla.obj-148371790. The bigger images are around 1 GB TiFF files. So for many images we have lots of detail. And some are huge, like a a panorama that is usualy a few metres wide. Using OSD together with IIP has been a massive win for us. We serve up over 1.3 million images using IIP (it's growing each week). Of those we use OSD to serve up around 280 000 images. The rest are served up using IIP into the internet archive's book reader as they are digitised books or ephemera that have been OCR'ed.

However the DZI technology has absolutely wowed our public users and stakeholders. We used the same digitised images - just converted into JPG2000, we digitised some of them 15 years ago, but the detail we can now show online is amazing.

Post up a URL to your application once it's complete so we can see what you have developed :)

@iangilman

[iangilman]

@terenceingram That's wonderful to hear! I love what you guys are doing!

[tommypreger]

I'm having this issue with a project I'm currently working on: https://urbanomonte-4db4a.web.app/
Currently it will not work on Safari or Chrome on my Iphone.

Interesting findings:

My openseadragon div is set to be 100% width and 100vh height, covering the whole window. If I make it smaller it seems to work, and if I then go into fullscreen it continues working! I would prefer not to use the fullscreen mode, but I'm curious as to why it works...

[msalsbery]

@tommypreger

Looks good on my iPhone, tested on Safari, Chrome, and Edge

[tommypreger]

@msalsbery

Thanks for testing! I came here to edit the post. I did some changes to the code and now it seems to work for me too.

I inspected the size of the canvas in Fullscreen mode vs my width/height 100% div and found that the Fullscreen canvas actually was a bit smaller, which probably is why it worked.

In my research I found out that 100vh includes the part of the screen hidden behind the bottom navigation bar on webkit browsers. I removed the CSS and used a resize javascript listener and resized the canvas to window.innerHeight instead, resulting in a slightly smaller height for my canvas, which made the difference.

[iangilman]

@tommypreger That's a really cool project! When you're ready to share it more widely, it would be great if you could add it to http://openseadragon.github.io/examples/in-the-wild/… Just make pull request against https://github.com/openseadragon/site-build/blob/master/www/in-the-wild.html to do so.