Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for HEIC/HEIF files #436

Open
thomasdn opened this issue Jul 7, 2021 · 23 comments
Open

Support for HEIC/HEIF files #436

thomasdn opened this issue Jul 7, 2021 · 23 comments

Comments

@thomasdn
Copy link
Contributor

thomasdn commented Jul 7, 2021

Newer Apple devices such as iPhone or iPads take photos in the HEIF format1. These photos are using the .heic extension.
Sigal does not seem to include the .heic files in the generated gallery.

Sigal seems to use the Pillow module for image manipulation. Pillow seems to support the HEIF format2, so I see no reason that sigal shouldn't automatically include .heic images present in the source folder.

@thomasdn
Copy link
Contributor Author

thomasdn commented Jul 9, 2021

I have been playing around with this a bit.
It seems that if I add .heic to img_extensions, then it fails in the following places:

The first immediate plate the user experiences an error is the following:

First, lots of these error during the "Collecting albums" step:

ERROR: Could not open image [...]img_6015.heic metadata: cannot identify image file '[...]img_6015.heic'                       
ERROR: Could not open image [...]img_6016.heic metadata: cannot identify image file '[...]img_6016.heic'                       
ERROR: Could not open image [...]img_6017.heic metadata: cannot identify image file '[...]img_6017.heic'                       
ERROR: Could not open image [...]img_6018.heic metadata: cannot identify image file '[...]img_6018.heic'                       
ERROR: Could not open image [...]img_6019.heic metadata: cannot identify image file '[...]img_6019.heic'                       
[...]

Then:

Traceback (most recent call last):
  File "[...]/sigal-dev/venv/bin/sigal", line 10, in <module>
    sys.exit(main())
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/sigal/__init__.py", line 157, in build
    gal.build(force=force)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/sigal/gallery.py", line 800, in build
    album_writer.write(album)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/sigal/writer.py", line 132, in write
    page = self.template.render(**context)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/sigal/themes/galleria/templates/album.html", line 1, in top-level template code
    {% extends "base.html" %}
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/sigal/themes/galleria/templates/base.html", line 18, in top-level template code
    {% block extra_head %}{% endblock extra_head %}
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/sigal/themes/galleria/templates/album.html", line 9, in block 'extra_head'
    content:url({{ album.medias[0].url }});
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/jinja2/environment.py", line 474, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: list object has no element 0

If sigal build is run with --debug, then it stops earlier producing this error:

$ sigal build --debug
[...]
INFO: Processing [...]/img_3862.heic


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "[...]/sigal-dev/venv/bin/sigal", line 10, in <module>
INFO: Failed to process: UnidentifiedImageError("cannot identify image file '[...]/img_3862.heic'")                                                                             
    sys.exit(main())
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/click/core.py", line 1137, in __call__
INFO: Failed to process: UnidentifiedImageError("cannot identify image file '[...]/img_3808.heic'")                                                                             
    return self.main(*args, **kwargs)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/sigal/__init__.py", line 157, in build
    gal.build(force=force)
  File "[...]/sigal-dev/venv/lib/python3.7/site-packages/sigal/gallery.py", line 756, in build
    for status in self.pool.imap_unordered(worker, media_list):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
PIL.UnidentifiedImageError: cannot identify image file '[...]/img_3334.heic'                                                                                                    
INFO: Processing [...]/img_3876.heic
INFO: Failed to process: UnidentifiedImageError("cannot identify image file '[...]/img_3876.heic'") 

Inside PIL: ./venv/lib/python3.7/site-packages/PIL/Image.py line 3023:

raise UnidentifiedImageError( 
    "cannot identify image file %r" % (filename if filename else fp)  
)  

It appears that it fails due to a problem in PIL.Image.

According to this stackoverflow post, it is possible to use pyheif to load HEIF/HEIC images with PIL.Image:
https://stackoverflow.com/questions/54395735/how-to-work-with-heic-image-file-types-in-python

The example looks like this:

 import whatimage
 import pyheif
 from PIL import Image


 def decodeImage(bytesIo):

    fmt = whatimage.identify_image(bytesIo)
    if fmt in ['heic', 'avif']:
         i = pyheif.read_heif(bytesIo)

         # Extract metadata etc
         for metadata in i.metadata or []:
             if metadata['type']=='Exif':
                 # do whatever

         # Convert to other file format like jpeg
         s = io.BytesIO()
         pi = Image.frombytes(
                mode=i.mode, size=i.size, data=i.data)

         pi.save(s, format="jpeg")

Would it make sense to use the same method to have sigal support HEIF images?

There are other methods proposed in the SO post that might be relevant.

@thomasdn
Copy link
Contributor Author

I think I almost fixed this using pyheif.

This is the code change I made to image.py:

--- venv2/lib/python3.7/site-packages/sigal/image.py    2021-07-07 11:49:55.452187776 +0200
+++ sigal/image.py      2021-07-10 14:39:09.921711241 +0200
@@ -48,6 +48,8 @@
 from . import signals, utils
 from .settings import Status, get_thumb
 
+import pyheif
+
 try:
     # Pillow 7.2+
     from PIL.TiffImagePlugin import IFDRational
@@ -70,7 +72,22 @@
     logger = logging.getLogger(__name__)
 
     with warnings.catch_warnings(record=True) as caught_warnings:
-        im = PILImage.open(file_path)
+        ext = os.path.splitext(file_path)[1]
+        if ext == ".heic":
+            try:
+                heif_file = pyheif.read(file_path)
+                im = PILImage.frombytes(
+                        heif_file.mode, 
+                        heif_file.size, 
+                        heif_file.data,
+                        "raw",
+                        heif_file.mode,
+                        heif_file.stride,
+                        )
+            except ValueError:
+                im = PILImage.open(file_path)
+        else:
+            im = PILImage.open(file_path)
 
     for w in caught_warnings:
         logger.warning(f'PILImage reported a warning for file {file_path}\n'

It now kinda works in that it actually reads the .heic images. However, when generating the thumbnails it fails because the thumbnails are now generated as jpeg format (which is correct) but the filename generated is still .heic (which is now incorrect as the file is no longer a heic file but a jpeg file).

I have inserted a hack with the try/except where I catch the ValueError. This ValueError is raised when sigal tries to generate thumbnails and opens a .heic file and then my conditional that checks if extension is ".heic" evaluates to True. However, the file is not a heif image. So pyheif fails causing the ValueError. My hack is then to just read the file normally using PILImage.open(). But, alas, this does not solve the file naming errors.

I think the file naming issues should be solvable in that they are very similar to how png files are handled. Here we also have different extensions and format. The difference is that a source PNG image will always be a PNG image. Even in the thumbnails.

I really hope that someone more experienced with this codebase will help implement this.

@saimn
Copy link
Owner

saimn commented Jul 10, 2021

https://pypi.org/project/pyheif-pillow-opener/ is mentioned in the Pillow issue, seems a nice and easy way.

@thomasdn
Copy link
Contributor Author

That does seem like an easy way to Just Fix it. Hopefully that'll work.
Not sure if it will solve the issue with filenames/thumbnails, though? Will these still have to be converted to jpegs to be shown on the web? And then the filenames would need to have different extensions, right?

@saimn
Copy link
Owner

saimn commented Jul 13, 2021

Hmm, for thumbnail extensions I don't remember exactly the status, there was an issue about this (#421)... There is still some confusion in the code between urls and filenames (#421 should have improved things but not sure if it's enough), and I also assumed (10 years ago 😄 ) that jpg was the main format. So there a bit of work I guess needed to fix that.

@thomasdn
Copy link
Contributor Author

Hmm, for thumbnail extensions I don't remember exactly the status, there was an issue about this (#421)... There is still some confusion in the code between urls and filenames (#421 should have improved things but not sure if it's enough), and I also assumed (10 years ago smile ) that jpg was the main format. So there a bit of work I guess needed to fix that.

Is this something you need help fixing? I am not a strong front-end dev, so not sure if I am the right person to take this on.

@thomasdn
Copy link
Contributor Author

I have tried replacing:

from PIL import Image as PILImage

With:

from PIL import Image as PILImage
from pyheif_pillow_opener import register_heif_opener
register_heif_opener()

In image.py and gallery.py.

The hope what that it would mostly work transparently. However, I get this exception:

INFO - Processing /home/sigal/sigal-dev/view_copy/2021__06/img_1738.heic
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/sigal/gallery.py", line 838, in worker
    return process_file(args)
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/sigal/gallery.py", line 833, in process_file
    return processor(media)
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/sigal/image.py", line 189, in process_image
    generate_image(media.src_path, media.dst_path, media.settings,
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/sigal/image.py", line 152, in generate_image
    save_image(img, outname, outformat, options=options, autoconvert=True)
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/pilkit/utils.py", line 203, in save_image
    save(wrapper)
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/pilkit/utils.py", line 191, in save
    img.save(fp, format, **options)
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/PIL/Image.py", line 2224, in save
    save_handler = SAVE[format.upper()]
KeyError: 'HEIF'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sigal/sigal-dev/venv/bin/sigal", line 8, in <module>
    sys.exit(main())
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/sigal/__init__.py", line 157, in build
    gal.build(force=force)
  File "/home/sigal/sigal-dev/venv/lib/python3.9/site-packages/sigal/gallery.py", line 758, in build
    for status in self.pool.imap_unordered(worker, media_list):
  File "/usr/lib/python3.9/multiprocessing/pool.py", line 870, in next
    raise value
KeyError: 'HEIF'

No sure how to proceed.

Full debug output is available here:
https://paste.yt/p15609.html

@saimn
Copy link
Owner

saimn commented Jul 20, 2021

Interesting, we cannot use this format for resized images and thumbnails, so saving should be forced to jpeg. But currently there is no real logic in the code to tell with output format is valid or not, and which formats should be converted. Fixing that is probably not that simple.

@thomasdn
Copy link
Contributor Author

Hi,

Could this be solved by having a Format class that describes an image format. Each format has a name, e.g. JPEG, PNG, etc. and a list of extensions which will match this format. Ie. if sigal encounters a file that matches a format's extension, then sigal will assume the file is in that format.
It will also have a specific out_format and out_extension. This will be whatever format sigal will use as output for this particular format. For JPEG the out_format willl also be JPEG. But maybe for a format like GIF the output format would be the free PNG format. Same for HEIF, the output format would be JPEG, etc.

Please see the code sample below to illustrate what I mean.

class Format:
    def __init__(self, name, extensions = [], out_format = None, out_extension = None):
        self.name = name
        self.extensions = extensions
        self.out_format = out_format
        self.out_extension = out_extension


jpg = Format("JPEG", ["jpg", "jpeg"], "JPEG", "jpg")
png = Format("PNG", ["png"], "PNG", "png")
gif = Format("GIF", ["gif"], "PNG", "png")
heif = Format("HEIF", ["heic"], "JPEG", "jpg")
webp = Format("WEBP", ["webp"], "JPEG", "jpg")

formats = [jpg, png, gif, heif, webp]

@saimn
Copy link
Owner

saimn commented Jul 21, 2021

I think we can have something simpler, and I don't want to maintain a list of which formats are supported by pillow. All we need is to define the formats that can be used for resized images and thumbnails, all the other would need to be converted. There is already a setting to force output images to a specific format, we probably need another one for the format to which images would be converted if there are not a valid output format.

# Output format of images (default: None, i.e. use input format)
img_format = "JPEG"

@bigcat88
Copy link

bigcat88 commented May 3, 2022

Is this enough, to support HEIF for Sigal?

plugins = [
    "sigal.plugins.adjust",
    "sigal.plugins.copyright",
    "sigal.plugins.extended_caching",
    "sigal.plugins.feeds",
    "sigal.plugins.nomedia",
    "sigal.plugins.watermark",
    "sigal.plugins.zip_gallery",
    "pillow_heif.HeifImagePlugin"         # <--- auto registration of Pillow plugin, from version 0.2.2
]

bigcat88/pillow_heif#22

@saimn
Copy link
Owner

saimn commented May 24, 2022

@bigcat88 - pillow_heif.HeifImagePlugin is a plugin for Pillow right ?

@bigcat88
Copy link

@saimn yes, it is one of two ways to register it as a plugin.
pillow-heif-registering-plugin

@saimn
Copy link
Owner

saimn commented Jun 8, 2022

@bigcat88 - so I guess you would need to make a plugin for sigal to register the Pillow plugin ?

@bigcat88
Copy link

bigcat88 commented Jun 8, 2022

@saimn no, i not need to, thanks. Do not want to add register function to module, cause that is library for HEIF and plugin for Pillow and not for anything else.
Just a suggestion(i am not a Sigal user):

for plugin in settings['plugins']:
        try:
            if isinstance(plugin, str):
                mod = importlib.import_module(plugin)
                mod.register(settings)  # Add here a check to not call `regsiter` when it is not present.
            else:
                plugin.register(settings)
            logger.debug('Registered plugin %s', plugin)
        except Exception as e:
            logger.error('Failed to load plugin %s: %r', plugin, e)

currently it works, just logs an error during register...

@saimn
Copy link
Owner

saimn commented Jun 10, 2022

Hmm ok but you're using sigal's plugin registration to register a Pillow plugin, relying on the fact that sigal is importing the plugin...
The config file is a Python file, so you can just import pillow_heif.HeifImagePlugin there.

@0-wiz-0
Copy link

0-wiz-0 commented Mar 8, 2023

So I tried extracting enough information from this to make a HEIF gallery, but I didn't get it to work.
I have sigal 2.3 and pillow_heif 0.10.0. I changed the default config to add:

import pillow_heif.HeifImagePlugin
img_format = "jpeg"
img_extensions = ['.jpg', '.jpeg', '.png', '.gif', '.heic']

I had a very weird error where PILImage.EXTENSION returns an empty dictionary, so some initialization seems missing? I don't know the proper fix for this, I hacked my way around this with the following patch to gallery.py:

--- sigal/gallery.py.orig       2022-04-08 18:19:56.000000000 +0000
+++ sigal/gallery.py
@@ -235,6 +235,7 @@ class Image(Media):
         super().__init__(filename, path, settings)
         imgformat = settings.get('img_format')
 
+        print(PILImage.registered_extensions())
         if imgformat and PILImage.EXTENSION[self.src_ext] != imgformat.upper():
             # Find the extension that should match img_format
             extensions = {v: k for k, v in PILImage.EXTENSION.items()}

Definitely not the right solution, I know, but after that, EXTENSION was not empty any longer and it proceeded a bit to finally fail with:

Collecting albums, done.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%      
Processing files  [------------------------------------]  0/177
Traceback (most recent call last):
  File "/usr/pkg/bin/sigal-3.11", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/sigal/__init__.py", line 174, in build
    gal.build(force=force)
  File "/usr/pkg/lib/python3.11/site-packages/sigal/gallery.py", line 853, in build
    for status in self.pool.imap_unordered(worker, media_list):
  File "/usr/pkg/lib/python3.11/multiprocessing/pool.py", line 873, in next
    raise value
  File "/usr/pkg/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/pkg/lib/python3.11/multiprocessing/connection.py", line 205, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle 'module' object

for which I don't even have a clue where to start fixing this. Help very welcome.

@bigcat88
Copy link

bigcat88 commented Mar 8, 2023

@0-wiz-0 can you test with pillow_heif==0.9.3 the same code?
I did not tested last version of pillow_heif(with native C extension) with multiprocessing module.

@0-wiz-0
Copy link

0-wiz-0 commented Mar 8, 2023

@bigcat88 : Fails the same way with version 0.9.3 of pillow_heif.

@bigcat88
Copy link

bigcat88 commented Mar 8, 2023

Just adding to config from PIL import Image without nothing more leads to this, so that is not a problem with pillow_heif

Simple solution to move import to image.py file(do not know how to do so when Sigal is installed, you can edit it in your venv for example):

# Force loading of truncated files
ImageFile.LOAD_TRUNCATED_IMAGES = True
from pillow_heif import HeifImagePlugin  # <---- add this line without comment ofc


def _has_exif_tags(img):
    return hasattr(img, "info") and "exif" in img.info

@0-wiz-0
Copy link

0-wiz-0 commented Mar 8, 2023

I'll try that, thanks.
This file has interesting code btw:

try:    
    # Pillow 7.2+
    from PIL.TiffImagePlugin import IFDRational
except ImportError: 
    IFDRational = None

I think it'd be good to do it the same way for heif support - try an import and if it fails, handle that (in the places that need it, if any).

@0-wiz-0
Copy link

0-wiz-0 commented Mar 8, 2023

Ok, so with:

--- sigal/gallery.py.orig       2022-04-08 18:19:56.000000000 +0000
+++ sigal/gallery.py
@@ -235,6 +235,7 @@ class Image(Media):
         super().__init__(filename, path, settings)
         imgformat = settings.get('img_format')
 
+       PILImage.registered_extensions()
         if imgformat and PILImage.EXTENSION[self.src_ext] != imgformat.upper():
             # Find the extension that should match img_format
             extensions = {v: k for k, v in PILImage.EXTENSION.items()}

--- sigal/image.py.orig 2022-04-08 18:19:56.000000000 +0000
+++ sigal/image.py
@@ -56,6 +56,11 @@ except ImportError:
 # Force loading of truncated files
 ImageFile.LOAD_TRUNCATED_IMAGES = True
 
+try:
+    from pillow_heif import HeifImagePlugin
+except ImportError:
+    HeifImagePlugin = None
+
 
 def _has_exif_tags(img):
     return hasattr(img, 'info') and 'exif' in img.info

and

img_extensions = ['.jpg', '.jpeg', '.png', '.gif', '.heic']
img_format = "jpeg"

in the config file, I get a gallery for HEIC images - yay!
Thanks, @bigcat88 for the hints :)

@thomasdn
Copy link
Contributor Author

in the config file, I get a gallery for HEIC images - yay! Thanks, @bigcat88 for the hints :)

Does this mean that your version is working with HEIC images in the source images and then having those presented in the gallery as jpeg?

If this works, will this be included in the next release of Sigal?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants