Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LibRaw 0.19 crashing DigiKam6beta on macOS Nikon RAW/NEF files #186

Closed
benjcmin opened this issue Sep 12, 2018 · 19 comments

Comments

Projects
None yet
3 participants
@benjcmin
Copy link

commented Sep 12, 2018

Hi,
I have an issue with LibRaw 0.19 crashing DigiKam6beta on macOS 10.13.6.
Already did some testing, it's not happening with an earlier build with LibRaw 0.18.11

This is the info from macOS crash report:
Thread 2 Crashed:: Digikam::ScanController
0 libsystem_platform.dylib 0x00007fff50d74c69 _platform_bzero$VARIANT$Haswell + 41
1 libdigikamcore.6.0.0.dylib 0x0000000106528cbb LibRaw::LibRaw(unsigned int) + 219
2 libdigikamcore.6.0.0.dylib 0x00000001064a0601 Digikam::DRawDecoder::rawFileIdentify(Digikam::RawInfo&, QString const&) + 641
3 libdigikamcore.6.0.0.dylib 0x00000001061123cf Digikam::RAWLoader::load(QString const&, Digikam::DImgLoaderObserver*) + 143
4 libdigikamcore.6.0.0.dylib 0x00000001063104b7 Digikam::DImg::load(QString const&, int, Digikam::DImgLoaderObserver*, Digikam::DRawDecoding const&) + 4743
5 libdigikamcore.6.0.0.dylib 0x000000010630f1ba Digikam::DImg::loadImageInfo(QString const&, bool, bool, bool, bool) + 298
6 libdigikamdatabase.6.0.0.dylib 0x000000010ed90e02 Digikam::ImageScanner::loadFromDisk() + 242
7 libdigikamdatabase.6.0.0.dylib 0x000000010ed912dc Digikam::ImageScanner::newFile(int) + 28
8 libdigikamdatabase.6.0.0.dylib 0x000000010ec9c657 Digikam::CollectionScanner::scanNewFile(QFileInfo const&, int) + 727
9 libdigikamdatabase.6.0.0.dylib 0x000000010ec97fc4 Digikam::CollectionScanner::scanAlbum(Digikam::CollectionLocation const&, QString const&) + 2548
10 libdigikamdatabase.6.0.0.dylib 0x000000010ec96634 Digikam::CollectionScanner::scanAlbumRoot(Digikam::CollectionLocation const&) + 180
11 libdigikamdatabase.6.0.0.dylib 0x000000010ec99fdb Digikam::CollectionScanner::partialScan(QString const&, QString const&) + 1627
12 libdigikamdatabase.6.0.0.dylib 0x000000010ec99922 Digikam::CollectionScanner::partialScan(QString const&) + 98
13 libdigikamgui.6.0.0.dylib 0x0000000104ec25e3 Digikam::ScanController::run() + 1795
14 org.qt-project.QtCore 0x000000010e7bb8f7 0x10e790000 + 178423
15 libsystem_pthread.dylib 0x00007fff50d7a661 _pthread_body + 340
16 libsystem_pthread.dylib 0x00007fff50d7a50d _pthread_start + 377
17 libsystem_pthread.dylib 0x00007fff50d79bf9 thread_start + 13

for reference DigiKam bug report https://bugs.kde.org/show_bug.cgi?id=398479

@benjcmin

This comment has been minimized.

Copy link
Author

commented Sep 14, 2018

From the DigiKam discussion I got the indication to run showfoto from the commandline and open a NEF file to get a better impression on whats happening.

running showfoto via terminal and then opening a NEF file via GUI.
On the command line I'm getting the following output:
digikam.widgets: Breeze icons resource file found
dbus[13232]: Dynamic session lookup supported but failed: launchd did not provide a socket path, verify that org.freedesktop.dbus-session.plist is loaded!
KMemoryInfo: Platform identified : "Unknown"
KMemoryInfo: TotalRam: -1
digikam.general: Allowing a cache size of 60 MB
digikam.geoiface: "setting backend marble"
digikam.showfoto: Switch to application font: QFont( ".SF NS Text,13,-1,5,50,0,0,0,0,0" )
digikam.widgets: Paths to color scheme : ("/opt/digikam/Applications/KF5/showfoto.app/Contents/Resources//digikam/colorschemes")
digikam.widgets: "" :: ""
QFSFileEngine::open: No file name specified
digikam.geoiface: "ROADMAP"
digikam.widgets: "" :: ""
digikam.showfoto: Switch to application font: QFont( ".SF NS Text,13,-1,5,50,0,0,0,0,0" )
digikam.showfoto: Switch to widget style: "Fusion"
digikam.geoiface: "ROADMAP"
QWidgetWindow(0x7ffc3b47f400, name="mainToolBarWindow") ( QScreen(0x7ffc38c77830, name="iMac") ): Attempt to set a screen on a child window.
QWidgetWindow(0x7ffc3b47f400, name="mainToolBarWindow") ( QScreen(0x7ffc38c77830, name="iMac") ): Attempt to set a screen on a child window.
QWidgetWindow(0x7ffc3b47f400, name="mainToolBarWindow") ( QScreen(0x7ffc38c77830, name="iMac") ): Attempt to set a screen on a child window.
QWidgetWindow(0x7ffc3b47f400, name="mainToolBarWindow") ( QScreen(0x7ffc38c77830, name="iMac") ): Attempt to set a screen on a child window.
digikam.general: file formats= ("BMP Bild (.bmp)", "CUR Bild (.cur)", "GIF Bild (.gif)", "ICNS Bild (.icns)", "ICO Bild (.ico)", "MNG Bild (.mng)", "PBM Bild (.pbm)", "PGM Bild (.pgm)", "PNG Bild (.png)", "PPM Bild (.ppm)", "SVG Bild (.svg)", "SVGZ Bild (.svgz)", "TGA Bild (.tga)", "WBMP Bild (.wbmp)", "WEBP Bild (.webp)", "XBM Bild (.xbm)", "XPM Bild (.xpm)", "TIFF-Bild (.tiff .tif)", "JPEG-Bild (.jpg *.jpeg .jpe)", "JPEG2000-Bild (.jp2 *.j2k .jpx .pgx)", "Progressive-Graphics-Datei (.pgf)", "Rohbilder (.bay *.bmq *.cr2 *.crw *.cs1 *.dc2 *.dcr *.dng *.erf *.fff *.hdr *.k25 *.kdc *.mdc *.mos *.mrw *.nef *.orf *.pef *.pxn *.raf *.raw *.rdc *.sr2 *.srf *.x3f *.arw *.3fr *.cine *.ia *.kc2 *.mef *.nrw *.qtk *.rw2 *.sti *.rwl .srw )", "Alle unterstützten Dateien (.bmp *.cur *.gif *.icns *.ico *.mng *.pbm *.pgm *.png *.ppm *.svg *.svgz *.tga *.wbmp *.webp *.xbm *.xpm *.tiff *.tif *.jpg *.jpeg *.jpe *.jp2 *.j2k *.jpx *.pgx *.pgf *.bay *.bmq *.cr2 *.crw *.cs1 *.dc2 *.dcr *.dng *.erf *.fff *.hdr *.k25 *.kdc *.mdc *.mos *.mrw *.nef *.orf *.pef *.pxn *.raf *.raw *.rdc *.sr2 *.srf *.x3f *.arw *.3fr *.cine *.ia *.kc2 *.mef *.nrw *.qtk *.rw2 *.sti *.rwl *.srw )")
objc[13232]: Class FIFinderSyncExtensionHost is implemented in both /System/Library/PrivateFrameworks/FinderKit.framework/Versions/A/FinderKit (0x7fff86108c90) and /System/Library/PrivateFrameworks/FileProvider.framework/OverrideBundles/FinderSyncCollaborationFileProviderOverride.bundle/Contents/MacOS/FinderSyncCollaborationFileProviderOverride (0x15129fcd8). One of the two will be used. Which one is undefined.
digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal => QDateTime(2016-02-14 16:38:45.000 CET Qt::TimeSpec(LocalTime))
digikam.metaengine: DateTime => Exif.Photo.DateTimeDigitized => QDateTime(2016-02-14 16:38:45.000 CET Qt::TimeSpec(LocalTime))
digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal => QDateTime(2016-02-14 16:38:45.000 CET Qt::TimeSpec(LocalTime))
digikam.metaengine: DateTime => Exif.Photo.DateTimeDigitized => QDateTime(2016-02-14 16:38:45.000 CET Qt::TimeSpec(LocalTime))
digikam.geoiface: ----
digikam.dimg: "/Users/benjcmin/Dropbox/tmp test/DSC_1796.NEF" : RAW file identified
digikam.general: Trying to load Embedded preview with libraw
Bus error: 10
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
Deleting expired sessions...28 completed.

[Prozess beendet]


And in the crash report from macOS I found the following

Thread 13 Crashed:: Thread (pooled)
0 libsystem_platform.dylib 0x00007fff50d74c69 _platform_bzero$VARIANT$Haswell + 41
1 libdigikamcore.6.0.0.dylib 0x0000000108125cbb LibRaw::LibRaw(unsigned int) + 219
2 libdigikamcore.6.0.0.dylib 0x000000010809b5e6 Digikam::DRawDecoder::loadEmbeddedPreview(QByteArray&, QString const&) + 630
3 libdigikamcore.6.0.0.dylib 0x000000010809a6e6 Digikam::DRawDecoder::loadEmbeddedPreview(QImage&, QString const&) + 54
4 libdigikamcore.6.0.0.dylib 0x00000001080414ea Digikam::ThumbnailCreator::createThumbnail(Digikam::ThumbnailInfo const&, QRect const&) const + 2122
5 libdigikamcore.6.0.0.dylib 0x000000010803f2c7 Digikam::ThumbnailCreator::load(Digikam::ThumbnailIdentifier const&, QRect const&, bool) const + 919
6 libdigikamcore.6.0.0.dylib 0x000000010803ef1f Digikam::ThumbnailCreator::load(Digikam::ThumbnailIdentifier const&) const + 79
7 libdigikamcore.6.0.0.dylib 0x0000000108050fe2 Digikam::ThumbnailLoadingTask::execute() + 1634
8 libdigikamcore.6.0.0.dylib 0x0000000108024424 Digikam::LoadSaveThread::run() + 372
9 libdigikamcore.6.0.0.dylib 0x00000001080244b9 non-virtual thunk to Digikam::LoadSaveThread::run() + 25
10 libdigikamcore.6.0.0.dylib 0x0000000108079773 Digikam::DynamicThread::Private::run() + 99
11 org.qt-project.QtCore 0x000000011001d6d3 0x10fff5000 + 165587
12 org.qt-project.QtCore 0x00000001100208f7 0x10fff5000 + 178423
13 libsystem_pthread.dylib 0x00007fff50d7a661 _pthread_body + 340
14 libsystem_pthread.dylib 0x00007fff50d7a50d _pthread_start + 377
15 libsystem_pthread.dylib 0x00007fff50d79bf9 thread_start + 13

Thread 13 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x000070000aaa61c8 rcx: 0x000000000004aa40 rdx: 0x000070000aaa61d0
rdi: 0x000070000aab0000 rsi: 0x0000000000000000 rbp: 0x000070000aaa6070 rsp: 0x000070000aaa6070
r8: 0x0000000000000000 r9: 0x000000000000800f r10: 0x0000000000000000 r11: 0x0000000000001e01
r12: 0x0000000108124dd0 r13: 0x0000000000000000 r14: 0x0000000000000000 r15: 0x000070000aaa61d0
rip: 0x00007fff50d74c69 rfl: 0x0000000000010202 cr2: 0x000070000aab0000

But honestly as I'm not a developer I can't judge if this information is helpfuel.

@cgilles

This comment has been minimized.

Copy link
Contributor

commented Sep 14, 2018

yes it is.

This want mean that crash appear in RAW preview extraction image. There is no demosaicing stage here.

The trace said : Digikam::DRawDecoder::loadEmbeddedPreview()

This method call a libraw one to get the JPEG preview image embeded in RAW data. The problem is here.

See the method code for details :

https://cgit.kde.org/digikam.git/tree/core/libs/rawengine/drawdecoder.cpp#n120

Gilles Caulier

@LibRaw

This comment has been minimized.

Copy link
Owner

commented Sep 18, 2018

According to backtrace, the crash occurs in bzero call, called by LibRaw::LibRaw

Could anyone with this crash provide exact LibRaw code line that crashes?
LibRaw::LibRaw(unsigned int) + 219 is not too informative

@benjcmin

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

Would love to help but as I wrote I'm not a developer and not sure how to provide this information.
I've you could give me some kind of intstruction how to provide the necessary information and trace the bug. Thx

@cgilles

This comment has been minimized.

Copy link
Contributor

commented Sep 19, 2018

It miss certainly the some debug symbols in the digiKam PKG installer even if i turn on all debug symbols at compilation time.

To libraw team : I think you can just try to reproduce the NEF jpeg extraction with libraw CLI tools under MacOS.

To DK user : can you share a NEF file which crash libraw under you MacOS ?

Gilles Caulier

@LibRaw

This comment has been minimized.

Copy link
Owner

commented Sep 19, 2018

@cgilles, we use LibRaw (including JPEG extraction) in our FastRawViewer and do not see any problem reports, so it the problem looks like digiKam specific.

@benjcmin

This comment has been minimized.

Copy link
Author

commented Sep 20, 2018

yes sure I could provide such a RAW file. Is it possible to add such a big file in here?
But I just opened the same file with Trial of FastRawViewer 1.4.9 and it didn't crash that tool. So maybe there is still something about DigiKam implementation of LibRaw.
And I saw there are some more LibRaw 0.19 macOS related topic on the LibRaw Form e.g.
Libraw 0.19 crashes on macOS if used in a background thread https://www.libraw.org/node/2414
Maybe this related somehow?

@LibRaw

This comment has been minimized.

Copy link
Owner

commented Sep 20, 2018

Is there any chance that stack size is limited with (newer) XCode?

@cgilles

This comment has been minimized.

Copy link
Contributor

commented Sep 20, 2018

Please use a cloud storage to host your RAW files...

Gilles Caulier

@benjcmin

This comment has been minimized.

Copy link
Author

commented Sep 20, 2018

@LibRaw

This comment has been minimized.

Copy link
Owner

commented Sep 26, 2018

Folks:
1st: the backtraces points to LibRaw::LibRaw (constructor), so the problem is not related to any specific RAW file

2nd: I've tried XCode 6, XCode 8, XCode 9 builds (using make -f Makefile.dist) and was unable to reproduce the problem using both dcraw_emu (single thread) and half_mt (multithreaded). Both samples works fine with DSC_1796.NEF sample (link above). The sample was multiplied into many (same) files: DSC_1796-[0-9].NEF

So, I'm still suspect that this is not LibRaw problem, but not enough stack problem. LibRaw objects are big (e.g. several 16-bit curves), so default stack size could be not enough (it is better to allocate LibRaw object dynamically to avoid that).

Is there any way to see is enough stack space is present in digiKam?

@LibRaw

This comment has been minimized.

Copy link
Owner

commented Sep 26, 2018

According to report from different user, this is stack size problem: https://www.libraw.org/comment/5057#comment-5057

LibRaw 0.19 object size is larger than 0.18, so please check this.

@cgilles

This comment has been minimized.

Copy link
Contributor

commented Sep 26, 2018

"Yes, LibRaw(0.19) object is significantly larger than (0.18). Looks like we need to move curve[0x1000] to dynamic allocation to save stack."

==> This imply what exactly ? We need to adjust something in the compilation/linking rules when libraw is built in digiKam core ? Did you apply something like that in libraw makefiles from official tarball ?

Gilles Caulier

@LibRaw

This comment has been minimized.

Copy link
Owner

commented Sep 26, 2018

You need to have enough stack space for your threads to hold LibRaw object or allocate LibRaw objects in threads dynamically.

What is thread stack size in digiKam?

@LibRaw

This comment has been minimized.

Copy link
Owner

commented Sep 26, 2018

And, answering second question:

  • there is no anything special in libraw build files
  • but multithreaded sample (half_mt) and our software (FastRawViewer) allocates LibRaw object on heap, not on stack.
@cgilles

This comment has been minimized.

Copy link
Contributor

commented Sep 26, 2018

What is thread stack size in digiKam? ==> never checked or set. So the response is "i don't know"
but multithreaded sample (half_mt) and our software (FastRawViewer) allocates LibRaw object on heap, not on stack. ==> This is certainly the solution. I will take a look.

Gilles Caulier

@cgilles

This comment has been minimized.

Copy link
Contributor

commented Sep 26, 2018

So the problem is to create the LibRaw object inside the fonction here :
https://cgit.kde.org/digikam.git/tree/core/libs/rawengine/drawdecoder.cpp#n106
The solution is to use a global LibRaw instance instead. Right ?
Gilles Caulier

@LibRaw

This comment has been minimized.

Copy link
Owner

commented Sep 26, 2018

Your code creates LibRaw on each thumbnail load.
You may use

  • global LibRaw object allocated in heap (via new). Make sure you never use same instance from different threads in the same time.
  • per-thread LibRaw object allocated in heap (via new LibRaw)
  • per-file LibRaw object allocated via 'new LibRaw'

per-thread long-living instance looks like better solution (you need to call recycle() to clean up allocations when not needed)

@cgilles

This comment has been minimized.

Copy link
Contributor

commented Sep 27, 2018

The problem is now fixed in digiKam with this commit :

https://commits.kde.org/digikam/031a951f0e9225834e3dfecc4d32389652c7b780

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.