fileid of oc_filecache in danger of hitting max value #26901

Open
schnidrig opened this Issue Jan 5, 2017 · 4 comments

Projects

None yet

4 participants

@schnidrig
Contributor

Database

5 node galera cluster with mariadb.
maxscale with read/write splitter.

Expected behaviour

virtually unlimited lifetime of oc service

Actual behaviour

the primary key of the filecache table is in danger of overflowing.
At that point our oc instance will refuse any addition of new files.

Problem description

MariaDB [owncloud]> show create table oc_filecache\G
*************************** 1. row ***************************
	   Table: oc_filecache
Create Table: CREATE TABLE `oc_filecache` (
  `fileid` int(11) NOT NULL AUTO_INCREMENT,
  `storage` int(11) NOT NULL DEFAULT '0',
  `path` varchar(4000) COLLATE utf8_bin DEFAULT NULL,
  `path_hash` varchar(32) COLLATE utf8_bin NOT NULL DEFAULT '',
  `parent` int(11) NOT NULL DEFAULT '0',
  `name` varchar(250) COLLATE utf8_bin DEFAULT NULL,
  `mimetype` int(11) NOT NULL DEFAULT '0',
  `mimepart` int(11) NOT NULL DEFAULT '0',
  `size` bigint(20) NOT NULL DEFAULT '0',
  `mtime` int(11) NOT NULL DEFAULT '0',
  `storage_mtime` int(11) NOT NULL DEFAULT '0',
  `encrypted` int(11) NOT NULL DEFAULT '0',
  `unencrypted_size` bigint(20) NOT NULL DEFAULT '0',
  `etag` varchar(40) COLLATE utf8_bin DEFAULT NULL,
  `permissions` int(11) DEFAULT '0',
  PRIMARY KEY (`fileid`),
  UNIQUE KEY `fs_storage_path_hash` (`storage`,`path_hash`),
  KEY `fs_parent_name_hash` (`parent`,`name`),
  KEY `fs_storage_mimetype` (`storage`,`mimetype`),
  KEY `fs_storage_mimepart` (`storage`,`mimepart`),
  KEY `fs_storage_size` (`storage`,`size`,`fileid`)
) ENGINE=InnoDB AUTO_INCREMENT=252069614 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
1 row in set (0.00 sec)

The primary key is a signed integer with max value of 2147483647.

MariaDB [owncloud]> select max(fileid), count(fileid) from oc_filecache;
+-------------+---------------+
| max(fileid) | count(fileid) |
+-------------+---------------+
|   252065705 |      60772002 |
+-------------+---------------+

We have 60 mio files and the fileid is currently at 250 mio.
4 months ago that counter was at 146 mio. During the last 4 months we had a 3 node galera cluster.
That means 100 Mio increase in 4 months; with our new 5 node cluster instead of a old 3 nodes cluster that would have been a: 100/3*5 = 170 mio increase. Therefore we'll hit the 2147483647 boundary in about 3.5 years with the current user base and setup. However, given the growth rate of our service, I'd estimate that to be more something like 2 years at best.

Considering, that enterprise customers (at least those I know of) usually are one year behind the current release, I think it is about time to fix this in the db.

Please consider changing the fileid to BIGINT. Or at the very least make it unsigned.

More of a hack than a workaround would be to change the auto_increment settings.

MariaDB [owncloud]> show variables like '%auto_increment%';
+------------------------------+-------+
| Variable_name                | Value |
+------------------------------+-------+
| auto_increment_increment     | 5     |
| auto_increment_offset        | 1     |
| wsrep_auto_increment_control | ON    |
+------------------------------+-------+

But of course, that would give me tons of other head aches.

@PVince81 PVince81 added this to the 10.0 milestone Jan 9, 2017
@PVince81
Collaborator
PVince81 commented Jan 9, 2017

@butonic this is the wrong ticket to discuss this, it will likely get lost. Maybe post here instead: #12487 (or find or create a matching ticket).

Yes for using bigint for fileid if that can fix the issue in this one ticket.

@Helios07
Helios07 commented Jan 9, 2017

It is even worse for us:

MariaDB [rwth_aachen]> select max(fileid), count(fileid) from oc_filecache;
+-------------+---------------+
| max(fileid) | count(fileid) |
+-------------+---------------+
|   529100321 |      51808811 |
+-------------+---------------+

(and we split our users to different instances already)

@PVince81
Collaborator
PVince81 commented Jan 9, 2017

We usually don't change DB schemas in a minor version, but might need to do an exception here... @felixboehm @DeepDiver1975 @butonic

I suggest to provide a PR for a quickfix that changes the column type to bigint (not sure about performance impact here during upgrade...) and address @butonic's other points separately.

@PVince81 PVince81 added the sev2-high label Jan 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment