Elodie crashes when non-ASCII character in filename. #379

filipfigzalski · 2020-07-07T16:32:52Z

I have folders with polish characters in name. When Elodie runs into them, I get this error:

Traceback (most recent call last):
  File "elodie.py", line 378, in <module>
    main()
  File "P:\Python\Python37-32\lib\site-packages\click\core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "P:\Python\Python37-32\lib\site-packages\click\core.py", line 696, in main
    rv = self.invoke(ctx)
  File "P:\Python\Python37-32\lib\site-packages\click\core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "P:\Python\Python37-32\lib\site-packages\click\core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "P:\Python\Python37-32\lib\site-packages\click\core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "elodie.py", line 141, in _import
    trash, allow_duplicates)
  File "elodie.py", line 68, in import_file
    media, allowDuplicate=allow_duplicates, move=False)
  File "c:\Users\Filip\Desktop\elodie-master\elodie\filesystem.py", line 525, in process_file
    metadata = media.get_metadata()
  File "c:\Users\Filip\Desktop\elodie-master\elodie\media\base.py", line 93, in get_metadata
    'date_taken': self.get_date_taken(),
  File "c:\Users\Filip\Desktop\elodie-master\elodie\media\photo.py", line 61, in get_date_taken
    exif = self.get_exiftool_attributes()
  File "c:\Users\Filip\Desktop\elodie-master\elodie\media\media.py", line 128, in get_exiftool_attributes
    self.exif_metadata = ExifTool().get_metadata(source)
  File "c:\Users\Filip\Desktop\elodie-master\elodie\external\pyexiftool.py", line 340, in get_metadata
    return self.execute_json(filename)[0]
  File "c:\Users\Filip\Desktop\elodie-master\elodie\external\pyexiftool.py", line 322, in execute_json
    return json.loads(self.execute(b"-j", *params).decode("utf-8"))
  File "P:\Python\Python37-32\lib\json\__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "P:\Python\Python37-32\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "P:\Python\Python37-32\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The text was updated successfully, but these errors were encountered:

crishnakh · 2020-11-06T21:07:47Z

I'm having this issue too.

xagrr · 2020-12-29T06:33:19Z

I'm not sure if I'm having quite the same issue but heres what it returns when I try to import from a "Masters" folder from an old iPhoto/Apple Photos library

/source/Masters/2014/09/12/20140912-103840/IMG_9457.JPG -> /destination/20140910/Apple-iPhone 5s/20140910-125554-unknown location-img_9457.jpg
Traceback (most recent call last):
  File "./elodie.py", line 378, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "./elodie.py", line 140, in _import
    dest_path = import_file(current_file, destination, album_from_folder,
  File "./elodie.py", line 67, in import_file
    dest_path = FILESYSTEM.process_file(_file, destination,
  File "/elodie/elodie/filesystem.py", line 543, in process_file
    directory_name = self.get_folder_path(metadata)
  File "/elodie/elodie/filesystem.py", line 364, in get_folder_path
    this_path = self.get_dynamic_path(part, mask, metadata)
  File "/elodie/elodie/filesystem.py", line 387, in get_dynamic_path
    folder = folder.replace(
TypeError: replace() argument 2 must be str, not int

jmathai · 2020-12-29T07:09:30Z

May be related to #291.

A couple of requests.

Do you know exactly which image it fails to import?
Can you paste the relevant output by also passing in the --debug flag?
What does your config.ini look like (appears you are customizing the file name)?

TypeError: replace() argument 2 must be str, not int

It might be helpful to see if casting the return of get_dynamic_path() on line 389.

str(self.get_dynamic_path(i[1:], i, metadata))

jmathai · 2020-12-29T07:11:01Z

@filipfigzalski and @crishnakh if you're still having this problem then please share more details as well.

xagrr · 2020-12-29T07:14:20Z

I have about 7000 files (not sure how many of them are pictures/movies) left in that directory and it processes 50-100 at a time before crashing like this. I'd left in the last file it tried to pull above. I've left a bit more info here with --debug on:

/source/Masters/2013/09/17/20130917-161815/._IMG_7222.JPG is not a valid media file. Skipping...
/source/.Trash-0/files/IMG_9231.JPG already at /destination/20140908/Apple-iPhone 5s/20140908-004349-unknown location-img_9231.jpg.
Original checksum returned None for /source/.Trash-0/files/IMG_9231.JPG. Skipping...
/source/.Trash-0/files/IMG_0644.JPG already at /destination/20130225/Apple-iPhone 3GS/20130225-112415-goa-img_0644.jpg.
Original checksum returned None for /source/.Trash-0/files/IMG_0644.JPG. Skipping...
/source/Masters/2014/06/21/20140621-125726/._IMG_8716.JPG is not a valid media file. Skipping...
/source/Masters/2013/06/07/20130607-231453/._IMG_6633.JPG is not a valid media file. Skipping...
/source/Masters/2013/03/05/20130305-221512/IMG_3238.JPG -> /destination/20130301/Apple-iPhone 4S/20130301-220501-gujarat-img_3238.jpg
/source/Masters/2013/11/27/20131127-193044/IMG_7434.JPG -> /destination/20131117/Apple-iPhone 4S/20131117-111232-philadelphia-pennsylvania-img_7434.jpg
Traceback (most recent call last):
  File "./elodie.py", line 378, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "./elodie.py", line 140, in _import
    dest_path = import_file(current_file, destination, album_from_folder,
  File "./elodie.py", line 67, in import_file
    dest_path = FILESYSTEM.process_file(_file, destination,
  File "/elodie/elodie/filesystem.py", line 543, in process_file
    directory_name = self.get_folder_path(metadata)
  File "/elodie/elodie/filesystem.py", line 364, in get_folder_path
    this_path = self.get_dynamic_path(part, mask, metadata)
  File "/elodie/elodie/filesystem.py", line 387, in get_dynamic_path
    folder = folder.replace(
TypeError: replace() argument 2 must be str, not int

config.ini

[MapQuest]
key=[_redacted_]
prefer_english_names=True

[Directory]
date=%Y%m%d
custom=%camera_make-%camera_model
full_path=%date/%custom

[File]
date=%Y%m%d-%H%M%S
location=%city-%state
name=%date-%location-%original_name-%album.%extension

edit: copied in the correct log
edit2: worth mentioning that I'm using this docker container and running this on an Unraid server. I couldn't get the Trash folder to map properly so it makes a hidden file inside my input folder.

jmathai · 2020-12-29T07:56:32Z

Thanks. A couple more things to help narrow down the file would be helpful.

Could you add this line here to log the file before it hits the error.

    for current_file in files:
        # *********** Add this line below ***********
        log.all('Preparing to process {}'.format(current_file))
        dest_path = import_file(current_file, destination, album_from_folder,
                    trash, allow_duplicates)

jmathai · 2021-01-09T22:45:43Z

I've not been able to replicate this issue. I can do some debugging if one of you can send me a zip file including the file which causes the error.

#396 seems to have had the problem be in the name of a folder - so a zip file which extracts the folder would be helpful there. // @angrycuban13

jmathai · 2021-01-11T07:22:08Z

Probably a duplicate of #251.

reykjavi · 2021-01-23T22:04:53Z

Hi,

I stumbled upon the same issue. I let myself to insert another debugging string to elodie code (that's all I can with Python :)) to identify a file and a path that brings trouble. Here's what I've got:

{"--- The current file is: ":"c:\photolib\photoarc\Фото из сада\Флешка 1\ФОТО\Средняя 2014-2015 гг\ПРАЗДНИК ОСЕНИ\DSCN0346.JPG"}
Traceback (most recent call last):
  File "./elodie.py", line 379, in <module>
    main()
  File "C:\Python\Python38\lib\site-packages\click\core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "C:\Python\Python38\lib\site-packages\click\core.py", line 696, in main
    rv = self.invoke(ctx)
  File "C:\Python\Python38\lib\site-packages\click\core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Python\Python38\lib\site-packages\click\core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python\Python38\lib\site-packages\click\core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "./elodie.py", line 141, in _import
    dest_path = import_file(current_file, destination, album_from_folder,
  File "./elodie.py", line 66, in import_file
    dest_path = FILESYSTEM.process_file(_file, destination,
  File "C:\repo\elodie\elodie\filesystem.py", line 524, in process_file
    metadata = media.get_metadata()
  File "C:\repo\elodie\elodie\media\base.py", line 93, in get_metadata
    'date_taken': self.get_date_taken(),
  File "C:\repo\elodie\elodie\media\photo.py", line 61, in get_date_taken
    exif = self.get_exiftool_attributes()
  File "C:\repo\elodie\elodie\media\media.py", line 128, in get_exiftool_attributes
    self.exif_metadata = ExifTool().get_metadata(source)
  File "C:\repo\elodie\elodie\external\pyexiftool.py", line 340, in get_metadata
    return self.execute_json(filename)[0]
  File "C:\repo\elodie\elodie\external\pyexiftool.py", line 322, in execute_json
    return json.loads(self.execute(b"-j", *params).decode("utf-8"))
  File "C:\Python\Python38\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Python\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python\Python38\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Also there is output of exiftool executable of that file:

ExifTool Version Number         : 12.16
File Name                       : DSCN0346.JPG
Directory                       : c:/photolib/photoarc/Фото из сада/Флешка 1/ФОТО/Средняя 2014-2015 гг/ПРАЗДНИК ОСЕНИ
Warning                         : FileName encoding not specified
File Size                       : 2.3 MiB
File Modification Date/Time     : 2014:10:29 11:26:06+03:00
File Access Date/Time           : 2021:01:24 00:54:23+03:00
File Creation Date/Time         : 2021:01:16 18:39:07+03:00
File Permissions                : rw-rw-rw-
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg
Exif Byte Order                 : Little-endian (Intel, II)
Image Description               :
Make                            : NIKON
Camera Model Name               : COOLPIX L23
Orientation                     : Horizontal (normal)
X Resolution                    : 300
Y Resolution                    : 300
Resolution Unit                 : inches
Software                        : COOLPIX L23    V1.1
Modify Date                     : 2014:10:29 11:26:05
Y Cb Cr Positioning             : Co-sited
Exposure Time                   : 1/30
F Number                        : 3.3
Exposure Program                : Program AE
ISO                             : 110
Sensitivity Type                : Standard Output Sensitivity
Exif Version                    : 0230
Date/Time Original              : 2014:10:29 11:26:05
Create Date                     : 2014:10:29 11:26:05
Components Configuration        : Y, Cb, Cr, -
Compressed Bits Per Pixel       : 2
Exposure Compensation           : 0
Max Aperture Value              : 2.7
Metering Mode                   : Multi-segment
Light Source                    : Unknown
Flash                           : Auto, Did not fire
Focal Length                    : 6.1 mm
Maker Note Version              : 2.00
Color Mode                      : Color
Quality                         : Normal
White Balance                   : Auto
Focus Mode                      : AF-S
Flash Setting                   : Normal
White Balance Fine Tune         : 0
ISO Selection                   : Auto
Compression                     : JPEG (old-style)
Preview Image Start             : 1834
Preview Image Length            : 36996
Image Processing                :
Face Detect Frame Size          : 320 240
Faces Detected                  : 0
Image Adjustment                : Normal
Manual Focus Distance           : 0.500011444
Digital Zoom                    : 1
AF Area Mode                    : Single Area
AF Point                        : Center
AF Points In Focus              : (none)
Scene Mode                      :
Saturation Adj                  : 0
Noise Reduction                 : Off
Scene Assist                    :
Date Stamp Mode                 : Off
Retouch History                 : None
Image Stabilization             : VR-Off
User Comment                    :
Flashpix Version                : 0100
Color Space                     : sRGB
Exif Image Width                : 3648
Exif Image Height               : 2736
Interoperability Index          : R98 - DCF basic file (sRGB)
Interoperability Version        : 0100
File Source                     : Digital Camera
Scene Type                      : Directly photographed
Custom Rendered                 : Normal
Exposure Mode                   : Auto
Digital Zoom Ratio              : 0
Focal Length In 35mm Format     : 43 mm
Scene Capture Type              : Standard
Gain Control                    : Low gain up
Contrast                        : Normal
Saturation                      : Normal
Sharpness                       : Normal
Subject Distance Range          : Unknown
Thumbnail Offset                : 60432
Thumbnail Length                : 2581
Image Width                     : 3648
Image Height                    : 2736
Encoding Process                : Baseline DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:2:2 (2 1)
Aperture                        : 3.3
Image Size                      : 3648x2736
Megapixels                      : 10.0
Preview Image                   : (Binary data 36996 bytes, use -b option to extract)
Scale Factor To 35 mm Equivalent: 7.0
Shutter Speed                   : 1/30
Thumbnail Image                 : (Binary data 2581 bytes, use -b option to extract)
Circle Of Confusion             : 0.004 mm
Field Of View                   : 45.4 deg
Focal Length                    : 6.1 mm (35 mm equivalent: 43.0 mm)
Hyperfocal Distance             : 2.65 m
Light Value                     : 8.2

Let me know if I can help you with any other details.

Thanks!

jmathai · 2021-01-24T04:15:02Z

Could you share the file with me via URL or email? jaisen (at) github-username (dot) com.

reykjavi · 2021-01-24T09:09:39Z

Could you share the file with me via URL or email? jaisen (at) github-username (dot) com.

Sent.

xagrr · 2021-01-24T19:54:45Z

I'm sorry I can't be responsive about this right now - I'm currently moving from NJ to Texas.

…

On Sun, Jan 24, 2021 at 4:09 AM reykjavi ***@***.***> wrote: Could you share the file with me via URL or email? jaisen (at) github-username (dot) com. Sent. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#379 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADCEWURRYW5LYBYDWRLMVETS3PPV5ANCNFSM4OS7WAIQ> .

reykjavi · 2021-04-04T22:25:45Z

Hey Jaisen,

I'm getting back to the issue as I'd like to help you with this and help other users as well.
Actually, the problem is about non-ascii characters as said in this issue. It can be a filename or a path that consists of such characters. For me it's the path name.
We had a debug session with you earlier, but we believed it's something about a file itself :)
I just checked my assumption by taking the erroneous file and copying it under C:\temp. Then elodie processed it just fine. The original path, however, has some cyrillic chars in it -- "c:\photolib\photoarc\Фото из сада\Флешка 1\ФОТО\Средняя 2014-2015 гг\ПРАЗДНИК ОСЕНИ\DSCN0346.JPG"

Huh!! May we hope you'll deal with this bug once? :)

jmathai · 2021-04-09T22:44:48Z

This bug indicates file name - but I think you're saying it's the path?

I'm curious if there's an underlying reason why non-ascii errors present themselves. We fixed a version of this in #167 andd a few other places.

xagrr · 2021-04-09T23:23:14Z

I’ll have to immerse myself back into this - most likely this summer. Sorry I haven’t been more of a help so far.

…

On Apr 9, 2021 at 6:45 PM, <Jaisen Mathai ***@***.***)> wrote: This bug indicates file name - but I think you're saying it's the path? I'm curious if there's an underlying reason why non-ascii errors present themselves. We fixed a version of this in #167 (#167) andd a few other places. — You are receiving this because you commented. Reply to this email directly, view it on GitHub (#379 (comment)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/ADCEWUU5T2J6PPMV5LZS2MLTH57O7ANCNFSM4OS7WAIQ).

reykjavi · 2021-04-11T16:26:29Z

@jmathai I believe it can be both, path or file (to OS it looks identical). I pulled the recent version of elodie, but the issue is still there.
You can reproduce it by creating the path I provided above.

xagrr · 2021-08-30T16:55:03Z

Did you still need help resolving this?

…

On 4/11/2021 12:26 PM, reykjavi wrote: @jmathai <https://github.com/jmathai> I believe it can be both, path or file (to OS it looks identical). I pulled the recent version of elodie, but the issue is still there. You can reproduce it by creating the path I provided above. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#379 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADCEWURNXGILEMITFPA2KULTIHEUHANCNFSM4OS7WAIQ>.

reykjavi · 2022-01-23T12:07:21Z

Hey,

I have moved to Mac recently and checked elodie with my photo archive again. Just to make you aware — there is no issue described in this topic on Mac. Therefore I can state it's Windows-specific issue.

crishnakh · 2022-01-23T12:17:45Z

Hi @reykjavi I have it on mac... so it's not only windows-specific issue.

jmathai mentioned this issue Jan 2, 2021

Traceback Error #396

Closed

jmathai added bug help wanted solution unknown labels Jan 11, 2021

jmathai added a commit that referenced this issue Jan 24, 2021

Add test to verify gh-379

1ec70ef

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elodie crashes when non-ASCII character in filename. #379

Elodie crashes when non-ASCII character in filename. #379

filipfigzalski commented Jul 7, 2020

crishnakh commented Nov 6, 2020

xagrr commented Dec 29, 2020

jmathai commented Dec 29, 2020 •

edited

Loading

jmathai commented Dec 29, 2020

xagrr commented Dec 29, 2020 •

edited

Loading

jmathai commented Dec 29, 2020

jmathai commented Jan 9, 2021

jmathai commented Jan 11, 2021

reykjavi commented Jan 23, 2021 •

edited by jmathai

Loading

jmathai commented Jan 24, 2021

reykjavi commented Jan 24, 2021

xagrr commented Jan 24, 2021 via email

reykjavi commented Apr 4, 2021 •

edited

Loading

jmathai commented Apr 9, 2021

xagrr commented Apr 9, 2021 via email

reykjavi commented Apr 11, 2021

xagrr commented Aug 30, 2021 via email

reykjavi commented Jan 23, 2022

crishnakh commented Jan 23, 2022

Elodie crashes when non-ASCII character in filename. #379

Elodie crashes when non-ASCII character in filename. #379

Comments

filipfigzalski commented Jul 7, 2020

crishnakh commented Nov 6, 2020

xagrr commented Dec 29, 2020

jmathai commented Dec 29, 2020 • edited Loading

jmathai commented Dec 29, 2020

xagrr commented Dec 29, 2020 • edited Loading

jmathai commented Dec 29, 2020

jmathai commented Jan 9, 2021

jmathai commented Jan 11, 2021

reykjavi commented Jan 23, 2021 • edited by jmathai Loading

jmathai commented Jan 24, 2021

reykjavi commented Jan 24, 2021

xagrr commented Jan 24, 2021 via email

reykjavi commented Apr 4, 2021 • edited Loading

jmathai commented Apr 9, 2021

xagrr commented Apr 9, 2021 via email

reykjavi commented Apr 11, 2021

xagrr commented Aug 30, 2021 via email

reykjavi commented Jan 23, 2022

crishnakh commented Jan 23, 2022

jmathai commented Dec 29, 2020 •

edited

Loading

xagrr commented Dec 29, 2020 •

edited

Loading

reykjavi commented Jan 23, 2021 •

edited by jmathai

Loading

reykjavi commented Apr 4, 2021 •

edited

Loading