occ Scan errors

[905jay]

Steps to reproduce

1. build a fresh VM and new install of docker nextcloud AIO from using docker and this github guide to a VM
2. perform all system updates to Debian 11
3. run through the initial startup of nextcloud-aio
4. create my users on the local system
5. rsync a large amount of data to the new user I created
6. On the host VM take ownership of all the files I rsync over
   chown -R www-data:www-data /var/lib/docker/volumes/nextcloud_aio_nextcloud_data/_data/jason/files
7. attempt to occ scan files for my user. Interestingly, it worked for other users on the system, but fails for my user "jason"
   nextcloud-aio:~$ sudo docker exec --user www-data -it 147c8ce311a6 php occ files:scan jason

Expected behavior

Running nextcloud from linuxserver.io on unraid, when adding a large amount of files I can use the occ command to have nextcloud recognize that they are now present.

Actual behavior

I can run the OCC command for all other users except jason

nextcloud-aio:$ sudo docker exec --user www-data -it 147c8ce311a6 php occ files:scan mike
Starting scan for user 1 out of 1 (mike)
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 0 | 0 | 00:00:00 |
+---------+-------+--------------+
nextcloud-aio:$ sudo docker exec --user www-data -it 147c8ce311a6 php occ files:scan nicole
Starting scan for user 1 out of 1 (nicole)
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 0 | 0 | 00:00:00 |
+---------+-------+--------------+
nextcloud-aio:$ sudo docker exec --user www-data -it 147c8ce311a6 php occ files:scan sandy
Starting scan for user 1 out of 1 (sandy)
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 0 | 0 | 00:00:00 |
+---------+-------+--------------+
nextcloud-aio:$ sudo docker exec --user www-data -it 147c8ce311a6 php occ files:scan andrew
Starting scan for user 1 out of 1 (andrew)
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 0 | 0 | 00:00:00 |
+---------+-------+--------------+

Host OS

Debian 11
Linux nextcloud-aio 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

Nextcloud AIO version

25.0.5

Current channel

Sysyem Details

Operating System: Linux 5.10.0-21-amd64 x86_64
CPU: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (4 cores)
Memory: 15.63 GB

PHP details

Version: 8.0.28
Memory limit: 512 MB
Max execution time: 3600
Upload max size: 10 GB
Extensions: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, ftp, hash, iconv, json, mbstring, SPL, PDO, pdo_sqlite, bz2, posix, readline, Reflection, session, SimpleXML, standard, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, cgi-fcgi, apcu, bcmath, Phar, exif, gd, gmp, imagick, imap, intl, ldap, memcached, pcntl, pdo_pgsql, pgsql, redis, smbclient, sodium, sysvsem, zip, libsmbclient, Zend OPcache

Database

Type: pgsql
Version: PostgreSQL 15.2 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
Size: 15.8 MB

Other valuable info

Starting scan for user 1 out of 1 (jason)
Exception during scan: An exception occurred while executing a query: SQLSTATE[25P02]: In failed sql transaction: 7 ERROR:  current transaction is aborted, commands ignored until end of transaction block
#0 /var/www/html/lib/private/DB/ConnectionAdapter.php(101): OC\DB\Exceptions\DbalException::wrap(Object(Doctrine\DBAL\Exception\DriverException))
#1 /var/www/html/lib/private/Files/Type/Loader.php(116): OC\DB\ConnectionAdapter->insertIfNotExist('*PREFIX*mimetyp...', Array)
#2 /var/www/html/lib/private/Files/Type/Loader.php(84): OC\Files\Type\Loader->store('text/rtf')
#3 /var/www/html/lib/private/Files/Cache/Cache.php(451): OC\Files\Type\Loader->getId('text/rtf')
#4 /var/www/html/lib/private/Files/Cache/Cache.php(299): OC\Files\Cache\Cache->normalizeData(Array)
#5 /var/www/html/lib/private/Files/Cache/Scanner.php(296): OC\Files\Cache\Cache->insert('files/Insight/S...', Array)
#6 /var/www/html/lib/private/Files/Cache/Scanner.php(226): OC\Files\Cache\Scanner->addToCache('files/Insight/S...', Array, -1)
#7 /var/www/html/lib/private/Files/Cache/Scanner.php(445): OC\Files\Cache\Scanner->scanFile('files/Insight/S...', 3, 4078, false, true, Array)
#8 /var/www/html/lib/private/Files/Cache/Scanner.php(391): OC\Files\Cache\Scanner->handleChildren('files/Insight/S...', true, 3, 4078, true, 113509)
#9 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Insight/S...', true, 3, 4078, true)
#10 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Insight/S...', true, 3, 3769, true)
#11 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Insight/S...', true, 3, 3549, true)
#12 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Insight', true, 3, 3484, true)
#13 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files', true, 3, 220, true)
#14 /var/www/html/lib/private/Files/Cache/Scanner.php(342): OC\Files\Cache\Scanner->scanChildren('', true, 3, 217, true, Array)
#15 /var/www/html/lib/private/Files/Utils/Scanner.php(256): OC\Files\Cache\Scanner->scan('', true, 3)
#16 /var/www/html/apps/files/lib/Command/Scan.php(161): OC\Files\Utils\Scanner->scan('/jason', true, NULL)
#17 /var/www/html/apps/files/lib/Command/Scan.php(217): OCA\Files\Command\Scan->scanFiles('jason', '/jason', false, Object(Symfony\Component\Console\Output\ConsoleOutput), false, true, false)
#18 /var/www/html/3rdparty/symfony/console/Command/Command.php(255): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#19 /var/www/html/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#20 /var/www/html/3rdparty/symfony/console/Application.php(1009): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#21 /var/www/html/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#22 /var/www/html/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#23 /var/www/html/lib/private/Console/Application.php(214): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#24 /var/www/html/console.php(100): OC\Console\Application->run()
#25 /var/www/html/occ(11): require_once('/var/www/html/c...')
#26 {main}
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 122     | 461   | 00:00:03     |
+---------+-------+--------------+

![2023-04-17 09 25 28](https://user-images.githubusercontent.com/13971036/232497604-c4c64f76-c18e-4a84-99b7-5efaae5508d1.png)```

[szaimen]

See https://help.nextcloud.com/t/error-while-running-occ-command-on-nextcloud-aio/153529

[905jay]

Hello, that link didn't provide any details that resolved the issue, as I had checked there previously before posting.
Permissions are correct as mentioned in the initial post, and are set to u+g www-data
I checked the DB logs and see this:

2023-04-17 11:11:04.241 EDT [3462] ERROR: invalid byte sequence for encoding "UTF8": 0x96

2023-04-17 11:11:04.241 EDT [3462] CONTEXT: unnamed portal parameter $8

2023-04-17 11:11:04.241 EDT [3462] STATEMENT: INSERT INTO "oc_filecache" ("mimepart", "mimetype", "mtime", "size", "etag", "storage_mtime", "permissions", "name", "parent", "checksum", "path_hash", "path", "storage") VALUES($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)

2023-04-17 11:11:04.253 EDT [3462] ERROR: current transaction is aborted, commands ignored until end of transaction block

2023-04-17 11:11:04.253 EDT [3462] STATEMENT: INSERT INTO "oc_filecache" ("mimepart", "mimetype", "mtime", "size", "etag", "storage_mtime", "permissions", "name", "parent", "checksum", "path_hash", "path", "storage") VALUES($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)

2023-04-17 11:11:04.257 EDT [3462] ERROR: current transaction is aborted, commands ignored until end of transaction block

2023-04-17 11:11:04.257 EDT [3462] STATEMENT: INSERT INTO "oc_filecache" ("mimepart", "mimetype", "mtime", "size", "etag", "storage_mtime", "permissions", "name", "parent", "checksum", "path_hash", "path", "storage") VALUES($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)

2023-04-17 11:11:04.262 EDT [3462] ERROR: current transaction is aborted, commands ignored until end of transaction block

2023-04-17 11:11:04.262 EDT [3462] STATEMENT: INSERT INTO "oc_filecache" ("mimepart", "mimetype", "mtime", "size", "etag", "storage_mtime", "permissions", "name", "parent", "checksum", "path_hash", "path", "storage") VALUES($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)

2023-04-17 11:11:04.267 EDT [3462] ERROR: current transaction is aborted, commands ignored until end of transaction block

2023-04-17 11:11:04.267 EDT [3462] STATEMENT: INSERT INTO "oc_mimetypes" ("mimetype") SELECT $1 FROM "oc_mimetypes" WHERE "mimetype" = $2 HAVING COUNT(*) = 0

[szaimen]

Sorry, the correct link is https://help.nextcloud.com/t/error-while-running-occ-command-on-nextcloud-aio/153529/6

[905jay]

Thanks for linking directly to the particular responses but that didn't provide an answer.
I provided a lot of detail on the issue, but if there is another detail I can provide that will help you and the community, please advise

[szaimen]

I just said in the link that I've posted that this likely happens due to non-UTF8 files being present on your file system. So you can try to rename these files or create a bug report in nextcloud/server where this needs to get fixed.

[905jay]

I am running a script to identify and convert all files in all sub folders to UTF-8
once that is complete, I will try the occ scan again, and report back here. Thank you

[905jay]

BTW for anyone wondering about the script I am running, here it is

I placed in on the VM that is hosting docker, and nextcloud-aio containers
the script is placed in the folder /var/lib/docker/volumes/nextcloud_aio_nextcloud_data/_data/

#!/bin/bash

# Find all non-UTF-8 files in current directory and subdirectories
files=$(find . -type f -exec file {} \; | grep -v "UTF-8" | cut -d: -f1)

# Loop through each file and convert it to UTF-8
for file in $files
do
  # Check if the file is a text file
  if [[ $(file "$file") == *"text"* ]]; then
    # Convert the file to UTF-8 using iconv
    iconv -f $(file -b --mime-encoding "$file") -t utf-8 "$file" -o "$file.utf8"
    # Replace the original file with the converted file
    mv "$file.utf8" "$file"
  fi
done

[905jay]

@szaimen can you recommend a way that I can convert those files, or identify which ones need to be renamed?
the script I pasted above ran and completed, and the issue remains.

[szaimen]

Please open an issue here: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

[szaimen]

Apparently https://help.nextcloud.com/t/will-not-be-accessible-due-to-incompatible-encoding/86612/10 might help