How to read multiple files properly?

[enrique-lozano]

Hi! First of all, thank you very much for this incredible project! I want to read multiple files at once, but the only way I can think of to do it is calling the read method multiple times in the following way:

function read(filteredFiles: string[]){
    const imgArray = [];
    
    for (const file of filteredFiles) {
      imgArray.push(exiftool.read(path.join(directoryPath, file)));
    }

    return Promise.all(imgArray);
}

However, this way the time to read each file is about 0.3 seconds! A very bad result considering that I want to read more than a thousand photos. Is there a better way to do this by prioritizing performance? Any way to extract only the needed data?

[enrique-lozano]

Other projects are also avoiding using exiftool due to ignorance or lack of documentation on the matter, the benchmarks are horrible compared to other libraries. Please see the benchmark created by the mantainer of pigallery2 here.

[mceachen]

Thanks for the heads-up!

I added some guidance to that thread you linked to, but here are a couple things to keep in mind:

1. By default, this library only spins up 1/4 the number of CPUs-worth of concurrency. If you're doing a benchmark, use something like new ExifTool({maxProcs: os.cpus().length}).
2. If you've got, say, 8 hyper-threaded cores, so you set maxProcs: 16, that should be fine on an SSD, but HDDs hate concurrency (especially high levels of concurrency), as the read-head may be physically butterflying across the entire platter continuously to get to all your different files. You may actually see faster performance with lower concurrency in this case.
3. Make sure you're measuring what you mean to measure: Promise resolution latency will be (roughly)
   (sum of latency for previously-enqueued-and-unsettled-requests / maxProcs) + latency-of-the-current-request
   This can skew your performance metrics if you aren't careful. (I've certainly made this mistake several times)

FWIW, I just ran the mktags script for this project, and on my AMD 5950x, Ubuntu 22, and a gen-3 SSD, overall read() throughput is ~500 files/second. This doesn't mean each read is 2ms, of course--this is with 25-30 subprocesses, so per-instance read() performance is more like 60ms.

[enrique-lozano]

Well, I'm not very interested in benchmarking as such, but rather in how I can use the library in the most efficient way possible in a real application and how it compares with others like exifr (see performance comparison). Anyway, thanks for your quick response! I will try to increase the number of cores to see if that works.

[mceachen]

Gotcha: well, I'd expect an embedded C library to pretty much always out-perform exiftool performance. I couldn't use it for PhotoStructure, though, as I found a heap of mimetypes and tags that exiftool provides and that I rely on that exifr didn't handle.

[enrique-lozano]

Yep, without a doubt the great advantage of this library is that exiftool is practically a standard in the photographic world, it is almost impossible for any other library to accept the amount of different tags and metadata that exiftool handles. The types of TS that you provide are also very neat! Again, thanks a lot for the help 😊