Failed to deserialize sjsir files on large project on scalajs 1.1.0

[ngbinh]

I see this error on a very large project at work when upgrading from scala.js 0.6.32 to 1.1.0

[info] Fast optimizing .../target/scala-2.13/scalajs-bundler/main/gondorappclient-fastopt.js
[debug] Linker: 57836 us
[error] java.io.IOException: Failed to deserialize /path-to/target/scala-2.13/classes/com/anduin/stargazer/fundsub/module/common/
FundSubZenDeskWidget$State.sjsir
[error]         at org.scalajs.linker.interface.unstable.IRFileImpl$$anonfun$withPathExceptionContext$1.applyOrElse(IRFileImpl.scala:65)
[error]         at org.scalajs.linker.interface.unstable.IRFileImpl$$anonfun$withPathExceptionContext$1.applyOrElse(IRFileImpl.scala:58)
[error]         at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
[error]         at scala.util.Failure.recover(Try.scala:234)
[error]         at scala.concurrent.Future.$anonfun$recover$1(Future.scala:395)
[error]         at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[error]         at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[error]         at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error]         at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1425)
[error]         at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) 
[error]         at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
[error]         at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
[error]         at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
[error]         at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
[error] Caused by: java.nio.file.FileSystemException: path-to-dir/FundSubZenDeskWidget$State.sjsir: Too many open files
[error]         at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
[error]         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
[error]         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
[error]         at java.base/sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:201)
[error]         at java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:253)
[error]         at java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:311)
[error]         at org.scalajs.linker.PathIRFile$PathIRFileImpl.$anonfun$withChannel$1(PathIRFile.scala:84)
[error]         at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[error]         at scala.util.Success.$anonfun$map$1(Try.scala:255)
[error]         at scala.util.Success.map(Try.scala:213)
[error]         at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[error]         at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[error]         at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[error]         at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error]         at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1425)
[error]         at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) 
[error]         at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
[error]         at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
[error]         at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
[error]         at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
[error] (gondorAppClient / Compile / fastOptJS) java.io.IOException: Failed to deserialize path-to-dir/FundSubZenDeskWidget$State.sjsir

chat log on gitter:

[gzm0]

The parts responsible for I/O do not throttle file opening whatsoever. So indeed the linker will try to open as many files as possible as fast as possible. That's probably causing this indeed.

[sjrd]

The only workaround I can think of--before we fix this--is to bump the maximum number of open files at the system level. Here is an article about that for Linux: https://linux.101hacks.com/unix/java-too-many-open-files/