Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

InnoDB: failed with error 95 #1

Closed
jbergstroem opened this issue Oct 27, 2018 · 19 comments · Fixed by #27
Closed

InnoDB: failed with error 95 #1

jbergstroem opened this issue Oct 27, 2018 · 19 comments · Fixed by #27
Labels
pending: upstream Issue is waiting for a third party to make progress type: bug Something isn't working as expected

Comments

@jbergstroem
Copy link
Owner

jbergstroem commented Oct 27, 2018

Money quote: InnoDB: preallocating 12582912 bytes for file ./ibdata1 failed with error 95

I've tried all "remedies" suggested to no avail:

innodb_flush_method = fsync
innodb_use_native_aio = false
lower_case_table_names = 2

A workaround is skipping innodb: docker -e SKIP_INNODB=1 ...

Upstream issues:

@jbergstroem
Copy link
Owner Author

It looks like the only workaround for the moment is avoiding mount binds. I will update readme to reflect this.

@jbergstroem
Copy link
Owner Author

I've added additional documentation to suggest using volumes. For those who would end up from other parts of the Internet, do something like:

$ docker volume create db
db
$ docker run -it --rm --name=db \
         -v db:/var/lib/mysql \
         jbergstroem/mariadb-alpine

More info can be found here.

I will keep this open until I can verify that mount binds will work with upstream mariadb/mysql.

@jbergstroem
Copy link
Owner Author

So, in more recent versions of MariaDB I can't reproduce this any longer. Will keep open a bit longer; but I still think a best practice is to create a volume as part of your workflow.

jbergstroem added a commit that referenced this issue Jun 21, 2020
Previous versions of MariaDB broke in certain scenarios
when using docker, aufs or other mounted drives with Mariadb `10.2.x`
and likely also some versions of `10.3.x`.

Fixes: #1
jbergstroem added a commit that referenced this issue Jun 21, 2020
Previous versions of MariaDB broke in certain scenarios
when using docker, aufs or other mounted drives with Mariadb `10.2.x`
and likely also some versions of `10.3.x`.

Fixes: #1
@ghost
Copy link

ghost commented Aug 11, 2020

Hey, I'm experiencing this again, also with SKIP_INNODB=1

@jbergstroem
Copy link
Owner Author

jbergstroem commented Aug 11, 2020

Hey, I'm experiencing this again, also with SKIP_INNODB=1

Hello! What version are you using? How are you starting it? Can you share all text output from docker? You should not be able to see an innodb-related error if you're skipping it.

@ghost
Copy link

ghost commented Aug 12, 2020

Hey, I'm experiencing this again, also with SKIP_INNODB=1

Hello! What version are you using? How are you starting it? Can you share all text output from docker? You should not be able to see an innodb-related error if you're skipping it.

Somehow when I tried it again today to send you logs it went through.. So everything is running now.. Yes, SKIP_INNODB=1 was used

@jbergstroem
Copy link
Owner Author

jbergstroem commented Aug 12, 2020

Somehow when I tried it again today to send you logs it went through.. So everything is running now.. Yes, SKIP_INNODB=1 was used

Ok – good to hear. Feel free to open a new issue should you have more problems.

@ghost
Copy link

ghost commented Aug 13, 2020

Thanks, however this InnoDB is broken problem persists.. It was just workaround.. You did not face this issue with latest build? I was running it on Debian 10.

@jbergstroem
Copy link
Owner Author

jbergstroem commented Aug 13, 2020

Thanks, however this InnoDB is broken problem persists.. It was just workaround.. You did not face this issue with latest build? I was running it on Debian 10.

I've tested on MacOS (my laptop) as well as running the test suite in github actions which runs Ubuntu 18.x. That should be a very similar environment to Debian 10 (glibc, kernel, packages, ..).

Can you provide me with the following and I will try to spin up a vm to test?

  1. What is your debian version? ($ cat /etc/debian_version)
  2. What is your docker version? ($ apt info docker-ce | head -n 2)
  3. What filesystem do you use to store docker volume/data in? (ZFS is known to have shown these symptoms - check the link in the issue)
  4. Provide output of: $ docker run -it jbergstroem/mariadb-alpine:10.4.13

Thanks!

@jbergstroem jbergstroem reopened this Aug 13, 2020
@jbergstroem jbergstroem changed the title 10.2.15: InnoDB is broken InnoDB: failed with error 95 Aug 13, 2020
@jbergstroem jbergstroem added the type: bug Something isn't working as expected label Aug 13, 2020
@jbergstroem
Copy link
Owner Author

I found your issue here: firefly-iii/firefly-iii#3688

Did you try working around the allocation issue by creating a volume in your docker compose and assigning it to this container (see above)? Should work.

I have a todo to create some docker-compose.yml examples while refactoring documentation. Thanks for reminding me.

@jbergstroem
Copy link
Owner Author

jbergstroem commented Aug 13, 2020

Answering myself after testing on a Debian 10.5 with ZFS:

Did you try working around the allocation issue by creating a volume in your docker compose and assigning it to this container (see above)? Should work.

It doesn't work:

Creating jbergstroem_fireflyiiidb_1 ... done
Creating jbergstroem_fireflyiii_1   ... done
Attaching to jbergstroem_fireflyiiidb_1, jbergstroem_fireflyiii_1
fireflyiiidb_1  | 2020-08-13 23:16:35 0 [ERROR] InnoDB: preallocating 12582912 bytes for file ./ibdata1 failed with error 95
fireflyiiidb_1  | 2020-08-13 23:16:35 0 [ERROR] InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space
fireflyiiidb_1  | 2020-08-13 23:16:35 0 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again.

Will compare to other alpines and see whats different; but last I dug into this, it was docker filesystem-related.

Edit: I have tried with yobasystems/alpine-mariadb which has the same issue.

@ghost
Copy link

ghost commented Aug 14, 2020

Hey ! Sorry that I did not reply faster. I really appreciate your care, hopefully it will get fixed.

@jbergstroem
Copy link
Owner Author

Money quote here:

Fallocate doesnt work on some storage backends used for Containers (LXD, Docker...). In my Case it was ZFS.

I'll investigate switching to another base.

@jbergstroem
Copy link
Owner Author

jbergstroem commented Sep 20, 2020

For those who are interested, here's a fairly technical writeup on one way to handle fallocate on COW filesystems (such as ZFS): openzfs/zfs#326 (comment)

@jbergstroem
Copy link
Owner Author

jbergstroem commented Sep 27, 2020

If you are monitoring this thread for updates: there is now a test container that's built on glibc via rawhide. It took a while to get right since I am a) building from source and b) moving into a SCRATCH container.

@jbergstroem
Copy link
Owner Author

Here's output from my zfs machine. Summary: the new container works whereas the old (still) breaks.

$ docker run -it -v test:/var/lib/mysql jbergstroem/mariadb-alpine:test-rawhide-glibc
2020-09-27 14:50:03 0 [Note] /bin/mysqld (mysqld 10.4.14-MariaDB) starting as process 1 ...
2020-09-27 14:50:03 0 [Note] InnoDB: Using Linux native AIO
2020-09-27 14:50:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-09-27 14:50:03 0 [Note] InnoDB: Uses event mutexes
2020-09-27 14:50:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-09-27 14:50:03 0 [Note] InnoDB: Number of pools: 1
2020-09-27 14:50:03 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-09-27 14:50:03 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2020-09-27 14:50:03 0 [Note] InnoDB: Initializing buffer pool, total size = 10M, instances = 1, chunk size = 10M
2020-09-27 14:50:03 0 [Note] InnoDB: Completed initialization of buffer pool
2020-09-27 14:50:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-09-27 14:50:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-09-27 14:50:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-09-27 14:50:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-09-27 14:50:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-09-27 14:50:03 0 [Note] InnoDB: Waiting for purge to start
2020-09-27 14:50:03 0 [Note] InnoDB: 10.4.14 started; log sequence number 60967; transaction id 21
2020-09-27 14:50:03 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-09-27 14:50:03 0 [Note] Plugin 'unix_socket' is disabled.
2020-09-27 14:50:03 0 [Note] InnoDB: Buffer pool(s) load completed at 200927 14:50:03
2020-09-27 14:50:03 0 [Note] Server socket created on IP: '::'.
2020-09-27 14:50:03 0 [Note] Reading of all Master_info entries succeeded
2020-09-27 14:50:03 0 [Note] Added new Master_info '' to hash table
2020-09-27 14:50:03 0 [Note] /bin/mysqld: ready for connections.
Version: '10.4.14-MariaDB'  socket: '/run/mysqld/mysqld.sock'  port: 3306  MariaDB Server
^C2020-09-27 14:50:19 0 [Note] /bin/mysqld (initiated by: unknown): Normal shutdown
2020-09-27 14:50:19 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-09-27 14:50:19 0 [Note] InnoDB: FTS optimize thread exiting.
2020-09-27 14:50:19 0 [Note] InnoDB: Starting shutdown...
2020-09-27 14:50:19 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2020-09-27 14:50:19 0 [Note] InnoDB: Instance 0, restricted to 156 pages due to innodb_buf_pool_dump_pct=25
2020-09-27 14:50:19 0 [Note] InnoDB: Buffer pool(s) dump completed at 200927 14:50:19
2020-09-27 14:50:20 0 [Note] InnoDB: Shutdown completed; log sequence number 60976; transaction id 22
2020-09-27 14:50:20 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-09-27 14:50:20 0 [Note] /bin/mysqld: Shutdown complete

$ docker run -it -v test2:/var/lib/mysql jbergstroem/mariadb-alpine:latest
2020-09-27 14:51:08 0 [ERROR] InnoDB: preallocating 12582912 bytes for file ./ibdata1 failed with error 95
2020-09-27 14:51:08 0 [ERROR] InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space
2020-09-27 14:51:08 0 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again.
2020-09-27 14:51:09 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-09-27 14:51:09 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-09-27 14:51:09 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2020-09-27 14:51:09 0 [ERROR] Aborting
<snip>

@svaroqui
Copy link

svaroqui commented Jan 4, 2021

Should be fixed on alpine see https://jira.mariadb.org/browse/MDEV-16015 can one provide feedback

@jbergstroem
Copy link
Owner Author

jbergstroem commented Jan 4, 2021

Should be fixed on alpine see https://jira.mariadb.org/browse/MDEV-16015 can one provide feedback

I noticed too! Thanks for the feedback - will try it (note: still not yet available through alpine 3.12 - so i have to build manually from their tree).

@jbergstroem
Copy link
Owner Author

This should be finally fixed as part of landing 10.4.18. Please test and reopen if so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pending: upstream Issue is waiting for a third party to make progress type: bug Something isn't working as expected
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants