Writing hangs forever

[maximeg]

Hi,

I've been encountering an issue with writing hanging forever. Nothing too special, it happened at first with the image processing pipeline of rails.

Vips is installed via Homebrew, on an up-to-date Mac Mini M1.

# vips --version    
vips-8.11.2-Sat Jul  3 14:17:10 UTC 2021

# vips --vips-config
enable debug: no
enable deprecated library components: yes
enable modules: no
use fftw3 for FFT: yes
accelerate loops with orc: yes
ICC profile support with lcms: yes (lcms2)
zlib: yes
text rendering with pangocairo: yes
font file support with fontconfig: yes
RAD load/save: yes
Analyze7 load/save: yes
PPM load/save: yes
GIF load:  yes
EXIF metadata support with libexif: yes
JPEG load/save with libjpeg: yes (pkg-config)
JXL load/save with libjxl: no (dynamic module: no)
JPEG2000 load/save with libopenjp2: yes
PNG load with libspng: yes
PNG load/save with libpng: yes (pkg-config libpng >= 1.2.9)
PNG quantisation to 8 bit: yes
TIFF load/save with libtiff: yes (pkg-config libtiff-4)
image pyramid save: yes
HEIC/AVIF load/save with libheif: yes (dynamic module: no)
WebP load/save with libwebp: yes
PDF load with PDFium:  no
PDF load with poppler-glib: yes (dynamic module: no)
SVG load with librsvg-2.0: yes
EXR load with OpenEXR: yes
OpenSlide load: yes (dynamic module: no)
Matlab load with matio: yes
NIfTI load/save with niftiio: no
FITS load/save with cfitsio: yes
Magick package: MagickCore (dynamic module: no)
Magick API version: magick7
load with libMagickCore: yes
save with libMagickCore: yes

Gem version: ruby-vips (2.1.2)

To reproduce, in rails console:

> Vips.version_string
"8.11.2-Sat Jul  3 14:17:10 UTC 2021"
> im = Vips::Image.jpegload("/Users/maxime/xxx/photo-1491933382434-500287f9b54b.jpg")
=> #<Image 1000x1250 uchar, 3 bands, srgb>
> im.write_to_file("/Users/maxime/xxx/test.jpg")
=> hang

Vips works in CLI:

vips rot photo-1491933382434-500287f9b54b.jpg x.jpg d90

Edit: Vips works in std irb:

> require "vips"
=> true
> Vips.version_string
"8.11.2-Sat Jul  3 14:17:10 UTC 2021"
> im = Vips::Image.jpegload("/Users/maxime/xxx/photo-1491933382434-500287f9b54b.jpg")
=> #<Image 1000x1250 uchar, 3 bands, srgb>
> im.write_to_file("/Users/maxime/xxx/test.jpg")
=> nil

[jcupitt]

Hi @maximeg, thanks for reporting this.

I think this is perhaps fork. My guess is:

• rails starts up and loads ruby-vips
• ruby-vips inits a range of thread resources to get ready for processing
• rails forks to handle a request
• the new process is a photocopy of the forking process, but all of the thread resources are broken, since you can't thread across fork
• ruby-vips now tries to process an image, but no thread stuff is working, and locks

We have a possible fix for this in 8.11.3 (as yet unreleased). If you are OK with editing your homebrew vips.rb formula, you could try that. Alternatively, 8.11.3 ought to be out fairly soon. You can also work around this by disabling spring (which will stop the rails fork behaviour).

[maximeg]

Hi @jcupitt

That is it :)

$ DISABLE_SPRING=true rails c
irb(main):001:0> Vips::Image.jpegload("/Users/maxime/xxx/photo-1491933382434-500287f9b54b.jpg").write_to_file("/Users/maxime/xxx/test.jpg")
=> nil

I use puma as a server, and it forks(
In prod, I still run 8.10 so I haven't the issue.

If you want me to test 8.11.3, I will with homebrew. Tell me what to put in the vips.rb :)

[jcupitt]

It's a bit fiddly -- if disabling spring gets you going, I'd just wait.

[wilg]

I'm having this error too – thought I was losing my mind until I found this issue!

I can repro/fix the same way with rails c, but I haven't been able to get Rails working again. I think I need to disable forking in Puma or something.

How's the fix looking? Was this issue introduced in a recent version that I could roll back before?

[jcupitt]

Yes, 8.11 has a new and faster threading system which starts up earlier in the libvips lifecycle. Rolling back to 8.10 should fix it, I think (or waiting for 8.11.3, or disabling fork. etc.)

[matpowel]

Just for future weary travelers, this was manifesting in our env as frozen tests. We run tests with parallelize turned on and tracking things down lead us to find it was VIPS causing the problem. Sure enough the problem goes away when you change the parallelize method to :threads so I think your theory about forking is correct @jcupitt, but that breaks a whole lot of other things like all system tests so not a solution.

So for now we have to turn parallelize off on our dev machines until 8.11.3 comes out I guess?

Luckily our CI is still running 8.10.1.

[matpowel]

PS. @jcupitt when do you expect 8.11.3 to be released?

[Nakilon]

Can't you split the tests into those that fork and have vips stubbed and those that use utilize vips with threads?

[wilg]

For those who may not be super familiar with the Ruby ecosystem I feel it's important to point out that forking is a super common thing for both production (puma web server and others) and development (spring) and tests (concurrent test runners) so disabling it long-term is very far from ideal.

Rolling back vips is very difficult with Homebrew also unfortunately.

But it sounds like a fix is upcoming so looking forward to that and thanks!

[matpowel]

@wilg os correct, this is not a problem with some obscure use case in Ruby, it's fundamental. I believe it's only not creating havoc in prod for us because we're on 8.10 there, otherwise since we use Puma multithreading so it'd be a disaster.

We won't be re-engineering our tests, we have loads that touch image processing and shouldn't be stubbed and would likely fail in thread mode anyway. It would be easier to replace Vips.

For now, the simple but annoying workaround is to disable Spring and parallelization in dev until the fix is out.

[jcupitt]

8.11.3 is out ow with (hopefully) a fix for this. Homebrew should update shortly.

https://github.com/libvips/libvips/releases/tag/v8.11.3

Sorry for the inconvenience this has caused.

[matpowel]

Wow, you're awesome @jcupitt, definitely didn't want to complain about the project, just wanted to be sure people knew it wasn't a trivial testing edge case only affected

[anitagraham]

Double plus 👍 for releasing that. I'm just starting to use ruby-vips and vips and I have been down a lot of rabbit holes trying to do things which should be quite simple. Off to download the new release now.

[jcupitt]

Hopefully this is fixed, I'll close. Please reopen if it's still broken.

[kevynlebouille]

Hi everyone,

I encounter some problem when I use ruby-vips with puma multi-workers.

Those lines appears in my console when I load a page from a Rails controller/view and a concurrent variant processing

objc[34825]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[34825]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

[jcupitt]

@kevynlebouille I think you might need this issue: #155

I'd check your libvips version.

[kevynlebouille]

@jcupitt yep, I saw this issue but it seems quite old now. I'm using a fresh setup with :

Mac OS : 11.6
xcode-select version : 2384
rbenv : 1.2.0
ruby-build : 20211019
libvips : vips-8.11.4

[jcupitt]

This is a different issue though -- this one is about threading breaking after fork, and I think it's resolved.

It looks like you've hit some kind of problem with Objective-C string init during a fork(), which might or might not be related to libvips. Could you move to #155, please?

[kevynlebouille]

I moved to #155. Thank you