Calendar import fails for clustered databases

[xteaun]

Steps to reproduce

1. Import a google calendar which is fairly large (mine has 800kb)

Expected behavior

Import all events from google calendar

Actual behaviour

Only imports some of the events

Calendar app version

4.2.4

CalDAV-clients used

nextcloud calendar app

Browser

Chrome Version 110.0.5481.100

Client operating system

MacOS Monterey 12.6.3 (21G419)

Server operating system

No response

Web server

Apache

Database engine version

MariaDB

PHP engine version

PHP 8.0

Nextcloud version

Nextcloud Hub 3 (25.0.4)

Updated from an older installed version or fresh install

Fresh install

List of activated apps

No response

Nextcloud configuration

No response

Web server error log

No response

Log file

No response

Browser log

No response

Additional info

No response

[jaylinski]

I had the same issue. Turns out the import feature sends a "PUT /remote.php/dav/calendars/user/personal/uuid.ics HTTP/2.0"-request via frontend for each calendar entry.

This caused my load-balancer to rate-limit/drop the import-requests.

A fix would be to do those API requests in the backend instead of sending them via the frontend.

[helmo]

I also noticed an issue during import ... 441 of the 469 events were imported.

In the logs I found:

{"Exception":"Doctrine\\\\DBAL\\\\Exception\\\\DeadlockException","Message":"An exception occurred while executing a query: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction","Code":1213,"Trace":[{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php","line":1943,
"function":"convert",
"class":"Doctrine\\\\DBAL\\\\Driver\\\\API\\\\MySQL\\\\ExceptionConverter",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
["Doctrine\\\\DBAL\\\\Query"]]},
{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php",
"line":1885,
"function":"handleDriverException",
"class":"Doctrine\\\\DBAL\\\\Connection",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
["Doctrine\\\\DBAL\\\\Query"]]},
{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php",
"line":1106,
"function":"convertExceptionDuringQuery",
"class":"Doctrine\\\\DBAL\\\\Connection",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
"SELECT `uid`,
 `displayname`,
 `password` FROM `oc_users` WHERE `uid_lower` = ?",
["exampleuser"],
[2]]},{"Exception":"Doctrine\\\\DBAL\\\\Exception\\\\DeadlockException","Message":"An exception occurred while executing a query: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction","Code":1213,"Trace":[{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php","line":1943,
"function":"convert",
"class":"Doctrine\\\\DBAL\\\\Driver\\\\API\\\\MySQL\\\\ExceptionConverter",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
["Doctrine\\\\DBAL\\\\Query"]]},
{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php",
"line":1885,
"function":"handleDriverException",
"class":"Doctrine\\\\DBAL\\\\Connection",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
["Doctrine\\\\DBAL\\\\Query"]]},
{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php",
"line":1106,
"function":"convertExceptionDuringQuery",
"class":"Doctrine\\\\DBAL\\\\Connection",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
"SELECT `uid`,
 `displayname`,
 `password` FROM `oc_users` WHERE `uid_lower` = ?",
["exampleuser"],
[2]]},

And:
"Message\":\"There is no active transaction\"

[helmo]

Splitting the file did not help ... both segments gave such errors.

• Import partially failed. Imported 223 out of 250.
• Import partially failed. Imported 195 out of 219.

[miaulalala]

I also noticed an issue during import ... 441 of the 469 events were imported.

In the logs I found:

{"Exception":"Doctrine\\\\DBAL\\\\Exception\\\\DeadlockException","Message":"An exception occurred while executing a query: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction","Code":1213,"Trace":[{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php","line":1943,
"function":"convert",
"class":"Doctrine\\\\DBAL\\\\Driver\\\\API\\\\MySQL\\\\ExceptionConverter",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
["Doctrine\\\\DBAL\\\\Query"]]},
{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php",
"line":1885,
"function":"handleDriverException",
"class":"Doctrine\\\\DBAL\\\\Connection",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
["Doctrine\\\\DBAL\\\\Query"]]},
{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php",
"line":1106,
"function":"convertExceptionDuringQuery",
"class":"Doctrine\\\\DBAL\\\\Connection",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
"SELECT `uid`,
 `displayname`,
 `password` FROM `oc_users` WHERE `uid_lower` = ?",
["exampleuser"],
[2]]},{"Exception":"Doctrine\\\\DBAL\\\\Exception\\\\DeadlockException","Message":"An exception occurred while executing a query: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction","Code":1213,"Trace":[{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php","line":1943,
"function":"convert",
"class":"Doctrine\\\\DBAL\\\\Driver\\\\API\\\\MySQL\\\\ExceptionConverter",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
["Doctrine\\\\DBAL\\\\Query"]]},
{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php",
"line":1885,
"function":"handleDriverException",
"class":"Doctrine\\\\DBAL\\\\Connection",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
["Doctrine\\\\DBAL\\\\Query"]]},
{"file":"/var/www/nextcloud_v29.0.2/nextcloud/3rdparty/doctrine/dbal/src/Connection.php",
"line":1106,
"function":"convertExceptionDuringQuery",
"class":"Doctrine\\\\DBAL\\\\Connection",
"type":"->",
"args":[["Doctrine\\\\DBAL\\\\Driver\\\\PDO\\\\Exception"],
"SELECT `uid`,
 `displayname`,
 `password` FROM `oc_users` WHERE `uid_lower` = ?",
["exampleuser"],
[2]]},

And: "Message\":\"There is no active transaction\"

Hm, that error doesn' really look related. Do you have debug mode enabled for your instance? If not, can you do so and try again? Then check the logfile for anything from "app":"dav"

[helmo]

I just tried to replicate it on a staging env with debug enabled, there the same ics file imported just fine.

Digging a bit deeper I temporarily sent all sql traffic to the first node in our MariaDB Galera cluster. That solved it for now.

Are the dav requests done in parallel, where they compete for the same lock in the database?

[miaulalala]

I just tried to replicate it on a staging env with debug enabled, there the same ics file imported just fine.

Digging a bit deeper I temporarily sent all sql traffic to the first node in our MariaDB Galera cluster. That solved it for now.

Are the dav requests done in parallel, where they compete for the same lock in the database?

Theoretically, requests should always go to the primary node and then replicate on the secondaries. But it definitely sounds suspicious that you were only able to reproduce this on the cluster and not staging, and only when not specifying the node.

[miaulalala]

Could you please tell me your installed apps and calendar app version @helmo ? It might not be the calendar app - there's events triggerd for the import as well, so the issue with clusters could be from a different part of the code.

If you have any logs that would be great as well!

[helmo]

This was on Nextcloud 29.0.2 and Calendar 4.7.6

[miaulalala]

I can't reproduce unfortunately; imported my very large calendar without issues. We did fix read- after- write issues on 29 so that is probably not it.

Can you observer the PUT requests in the network tab when importing in calendar and copy the response for a failing PUT request?

[helmo]

I did, and can see 500 errors for PUT requests, with two different responses:

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Doctrine\DBAL\Driver\PDO\PDOException</s:exception>
  <s:message>There is no active transaction</s:message>
</d:error>

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>OC\DB\Exceptions\DbalException</s:exception>
  <s:message>An exception occurred while executing a query: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction</s:message>
</d:error>

And a few 404's:

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\NotFound</s:exception>
  <s:message>Calendar object not found</s:message>
</d:error>

These are evenly distributed over the 400+ requests,