Are thumbnails used for face recognition?

[JanisPlayer]

I've been having problems with facial recognition for a while. I noticed that unsupported formats appear in the console every now and then. I use a Nextcloud function that only comes in version 28 and am asking to avoid problems with the app. This extension ensures that the preview generator with imaginary previews generated in WebP. In any case, this is enforced for all images if you have WebP activated as a format. I suspect that leads to problems. I noticed the generator is also used by this app. And every now and then I get format support errors with JPEG images. Some detections then work again. Are the previews used as a basis for recognition? Then it would explain why only images with JPEG preview are recognized and scanned for me. I noticed the problem by accident, I've been wondering what was going on for months. Maybe you can help me.

[marcelklehr]

Hey @JanisPlayer
Yes, recognize uses the preview generator to scale images down for recognition. Currently, only the following formats are supported for classification: https://github.com/nextcloud/recognize/blob/main/lib/Constants.php#L10

[JanisPlayer]

Thank you for the response, now I know where the problem lies. In the getConvertedFilePath() function, it seems that only .jpg is appended. Since I've set 'preview_format' => 'webp', in Imaginary, this format is enforced by Imaginary.

Recognize uses the Nextcloud Preview Generator, and it has no way to influence the output format. This leads to an error in recognize/src/classifier_faces.js:56:28 because this format is not supported.

So, there are a few potential solutions:

1. Check if the available preview image is in an unsupported format and then create a new one in a supported format. To create a new image in a desired format, you may need to rewrite the previewProvider so that an app can specify the format and perhaps even the quality. This solution could potentially improve recognition accuracy, but it has the drawback that the Nextcloud previewProvider would need to be modified, which might not happen as quickly as needed.

2. Add support for WebP in classifier_faces.js and hope that it doesn't require extensive changes.

3. Implement a custom preview function for Recognize, which could be a quick and easy solution with potential advantages in recognition quality, as the quality settings of JPEG would no longer be a concern.

These are the options that came to mind quickly to solve the problem. You can find the code related to the issue here:

recognize/lib/Classifiers/Classifier.php

Line 248 in 9b8296b

private function getConvertedFilePath(Node $file): string {

		// Create a temporary file *with the correct extension*
		$tmpname = $this->tempManager->getTemporaryFile('.jpg');

		if (!$this->previewProvider->isAvailable($file) && !($file instanceof File)) { // Need if not support format.
			return $path;
		}

		try {
			$this->logger->debug('generating preview of ' . $file->getId() . ' with dimension '.self::TEMP_FILE_DIMENSION);
			$image = $this->previewProvider->getPreview($file, self::TEMP_FILE_DIMENSION, self::TEMP_FILE_DIMENSION); // Need format and quality output prameter. Or a own previewProvider function.
		} catch(\Throwable $e) {
			$this->logger->info('Failed to generate preview of ' . $file->getId() . ' with dimension '.self::TEMP_FILE_DIMENSION . ': ' . $e->getMessage());
			return $path;
		}

Error:

Result for /janis/files/SofortUpload/Takeout/Screenshot_20191201-151844.png(oc_tmp_ChP37O-.jpg) = []
Face results for 4119864 are in
Classifier process output: Error: Expected image (BMP, JPEG, PNG, or GIF), but got unsupported image type
    at getImageType (/var/www/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/image.js:353:15)
    at Object.decodeImage (/var/www/nextcloud/apps/recognize/node_modules/@tensorflow/tfjs-node/dist/image.js:198:21)
    at main (/var/www/nextcloud/apps/recognize/src/classifier_faces.js:56:28)

[JanisPlayer]

I've conducted an experiment using GD to generate 100% JPEGs for the AI. The result is that everything works much better. However, I believe there's currently a missing option to utilize the RAM disk. 100% quality might also be a bit excessive; 95% should suffice.

I'm more inclined to rewrite the Nextcloud Preview Generator in a way that allows controlling the format and quality of images through the API. Since my current patches enable users to do this, it would be straightforward to adapt them for app developers to control the output.

The advantage is that AI apps won't have issues with poor user settings, reducing complaints about error recognition and making everyone happier.

It would also be helpful to mention in the documentation that the previews are used for AI, as this caused issues for me.

Here is my test code modification. I can, of course, refactor this into its own function or insert it more cleanly:

		if (!$this->previewProvider->isAvailable($file) && !($file instanceof File)) {
			return $path;
		}

		try {
			$this->logger->debug('generating preview of ' . $file->getId() . ' with dimension '.self::TEMP_FILE_DIMENSION);
			//$image = $this->previewProvider->getPreview($file, self::TEMP_FILE_DIMENSION, self::TEMP_FILE_DIMENSION);

			$imagetype = exif_imagetype($path); //To troubleshoot console errors, GD does not support all formats.
			$image_read = false;
		  if (0 < $imagetype) {
				$image = imagecreatefromstring(file_get_contents($path));
				$width = imagesx($image);
				$height = imagesy($image);

				$maxWidth = self::TEMP_FILE_DIMENSION;
				$maxHeight = self::TEMP_FILE_DIMENSION;

				if ($width > $maxWidth || $height > $maxHeight) {
					$aspectRatio = $width / $height;
					if ($width > $height) {
							$newWidth = $maxWidth;
							$newHeight = $maxWidth / $aspectRatio;
					} else {
							$newHeight = $maxHeight;
							$newWidth = $maxHeight * $aspectRatio;
					}
						$image = imagescale($image, (int)$newWidth, (int)$newHeight);
				}
				// Is the same like Imagick
				// $image = new Imagick();
				// $image->readImage($path);
				// $image->setImageFormat("jpeg");
				// $image->setImageCompressionQuality(100);
				// $image->scaleImage(self::TEMP_FILE_DIMENSION, self::TEMP_FILE_DIMENSION, true);
			} else {
				$image = $this->previewProvider->getPreview($file, self::TEMP_FILE_DIMENSION, self::TEMP_FILE_DIMENSION);
				try {
					$preview = $image->read();
				} catch (NotPermittedException $e) {
					$this->logger->warning('Could not read preview file', ['exception' => $e]);
				}

				if ($preview === false) {
					$this->logger->warning('Could not open preview file');
					return $path;
				}
		  	$tmpfile = fopen($tmpname, 'wb');
				if (stream_copy_to_stream($preview, $tmpfile) === false) {
					$this->logger->info('Could not copy preview file to temp folder');
					fclose($preview);
					fclose($tmpfile);
				}
				fclose($preview);
				fclose($tmpfile);

				// $imageInfo = getimagesizefromstring($imageString);
				// if ($imageInfo !== false) {
				//     $imagetype = $imageInfo[2];
				// } else {
				// 		$imagetype = false;
				// }

				$imagetype = exif_imagetype($tmpname);

				if (in_array($imagetype, [IMAGETYPE_WEBP, IMAGETYPE_AVIF, false])) { // To troubleshoot if it is a webp or avif.
					$image = imagecreatefromstring(file_get_contents($tmpname));
					unlink($tmpname);
			  	} else {
					//$image = $image->read(); //fclose($preview); need a if or return $tmpname; or try.
					$image_read = true;
				}
			}
		} catch(\Throwable $e) {
			$this->logger->info('Failed to generate preview of ' . $file->getId() . ' with dimension '.self::TEMP_FILE_DIMENSION . ': ' . $e->getMessage());
			return $path;
		}


		if ($image_read == true) {
			// try {
			// 	$preview = $image->read();
			// } catch (NotPermittedException $e) {
			// 	$this->logger->warning('Could not read preview file', ['exception' => $e]);
			// 	return $path;
			// }
			//
			// if ($preview === false) {
			// 	$this->logger->warning('Could not open preview file');
			// 	return $path;
			// }
			//
			// $this->logger->debug('Copying ' . $file->getId() . ' preview to tempfolder');
			//
			// if (stream_copy_to_stream($preview, $tmpfile) === false) {
			// 	$this->logger->warning('Could not copy preview file to temp folder');
			// 	fclose($preview);
			// 	fclose($tmpfile);
			// 	return $path;
			// }
			// fclose($preview);
			// fclose($tmpfile);

			return $tmpname;
		} else {
			// code...
		}

		$tmpfile = fopen($tmpname, 'wb');

		if ($tmpfile === false) {
			$this->logger->warning('Could not open tmpfile');
			return $path;
		}

		try {
			//$preview = $image->read();
			$preview = $image;
		} catch (NotPermittedException $e) {
			$this->logger->warning('Could not read preview file', ['exception' => $e]);
			return $path;
		}

		if ($preview === false) {
			$this->logger->warning('Could not open preview file');
			return $path;
		}

		$this->logger->debug('Copying ' . $file->getId() . ' preview to tempfolder');

		//if (stream_copy_to_stream($preview, $tmpfile) === false) {
		if (imagejpeg($preview, $tmpfile, 100) === false) {
			$this->logger->warning('Could not copy preview file to temp folder');
			//fclose($preview);
			fclose($tmpfile);
			return $path;
		}
		//fclose($preview);
		fclose($tmpfile);

		return $tmpname;
	}