Installation problem with 1.3.0 and 1.4.0-rc1 -- MariaDB 10.2.10 issue on ext3

[bergtwvd]

I follow the install/config instructions from the book. When I start it all up with install.sh I notice a problem in the database. Same problem for 1.3.0 and 1.4.0-rc1. Details below.

When I place the volumes of the db and registry in a 75 Gb filesystem, problem persists.

InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space

There is enough disk space. On what filesystem does ibdata reside?

Any idea of the cause and solution?

1)Version of docker engine and docker-compose.

Docker version 1.11.2, build b9f10c9
docker-compose version 1.7.1, build 0a9ab35

2)Config files of harbor, you can get them by packaging "harbor.cfg" and files in the same directory, including subdirectory.

## Configuration file of Harbor

#The IP address or hostname to access admin UI and registry service.
#DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname = <snip/>

#The protocol for accessing the UI and token/notification service, by default it is http.
#It can be set to https if ssl is enabled on nginx.
ui_url_protocol = https

#The password for the root user of mysql db, change this before any production use.
db_password = root123

#Maximum number of job workers in job service
max_job_workers = 3

#Determine whether or not to generate certificate for the registry's token.
#If the value is on, the prepare script creates new root cert and private key
#for generating token to access the registry. If the value is off the default key/cert will be used.
#This flag also controls the creation of the notary signer's cert.
customize_crt = off

#The path of cert and key files for nginx, they are applied only the protocol is set to https
ssl_cert = /home/xxx/certs/domain.crt
ssl_cert_key = /home/xxx/certs/domain.key

#The path of secretkey storage
secretkey_path = /data

#Admiral's url, comment this attribute, or set its value to NA when Harbor is standalone
admiral_url = NA

#The password of the Clair's postgres database, only effective when Harbor is deployed with Clair.
#Please update it before deployment, subsequent update will cause Clair's API server and Harbor unable to access Clair's database.
clair_db_password = password

#Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
log_rotate_count = 50
#Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
#If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
#are all valid.
log_rotate_size = 200M

#NOTES: The properties between BEGIN INITIAL PROPERTIES and END INITIAL PROPERTIES
#only take effect in the first boot, the subsequent changes of these properties
#should be performed on web ui

#************************BEGIN INITIAL PROPERTIES************************

#Email account settings for sending out password resetting emails.

#Email server uses the given username and password to authenticate on TLS connections to host and act as identity.
#Identity left blank to act as username.
email_identity =

email_server = mailhost2a.tno.nl
email_server_port = 25
email_username =
email_password =
email_from = Harbor admin <snip/>
email_ssl = false
email_insecure = false

##The initial password of Harbor admin, only works for the first time when Harbor starts.
#It has no effect after the first launch of Harbor.
#Change the admin password from UI after launching Harbor.
harbor_admin_password = Harbor12345

##By default the auth mode is db_auth, i.e. the credentials are stored in a local database.
#Set it to ldap_auth if you want to verify a user's credentials against an LDAP server.
auth_mode = db_auth

#The url for an ldap endpoint.
ldap_url = ldaps://ldap.mydomain.com

#A user's DN who has the permission to search the LDAP/AD server.
#If your LDAP/AD server does not support anonymous search, you should configure this DN and ldap_search_pwd.
#ldap_searchdn = uid=searchuser,ou=people,dc=mydomain,dc=com

#the password of the ldap_searchdn
#ldap_search_pwd = password

#The base DN from which to look up a user in LDAP/AD
ldap_basedn = ou=people,dc=mydomain,dc=com

#Search filter for LDAP/AD, make sure the syntax of the filter is correct.
#ldap_filter = (objectClass=person)

# The attribute used in a search to match a user, it could be uid, cn, email, sAMAccountName or other attributes depending on your LDAP/AD
ldap_uid = uid

#the scope to search for users, 0-LDAP_SCOPE_BASE, 1-LDAP_SCOPE_ONELEVEL, 2-LDAP_SCOPE_SUBTREE
ldap_scope = 2

#Timeout (in seconds)  when connecting to an LDAP Server. The default value (and most reasonable) is 5 seconds.
ldap_timeout = 5

#Verify certificate from LDAP server
ldap_verify_cert = true

#Turn on or off the self-registration feature
self_registration = on

#The expiration time (in minute) of token created by token service, default is 30 minutes
token_expiration = 30

#The flag to control what users have permission to create projects
#The default value "everyone" allows everyone to creates a project.
#Set to "adminonly" so that only admin user can create project.
project_creation_restriction = adminonly

#The follow configurations are for Harbor HA mode only
#####################################################
#the address of the mysql database.
db_host = mysql

#The port of mysql database host
db_port = 3306

#The user name of mysql database
db_user = root
#The redis server address
redis_url =

#Clair DB host address
clair_db_host = postgres

#Clair DB connect port
clair_db_port = 5432

#Clair DB username
clair_db_username = postgres

#Clair default database
clair_db = postgres
################### end of HA section #####################

#************************END INITIAL PROPERTIES************************

#The following attributes only need to be set when auth mode is uaa_auth
uaa_endpoint = uaa.mydomain.org
uaa_clientid = id
uaa_clientsecret = secret
uaa_verify_cert = true
uaa_ca_cert = /path/to/ca.pem


### Docker Registry setting ###
#registry_storage_provider can be: filesystem, s3, gcs, azure, etc.
registry_storage_provider_name = filesystem
#registry_storage_provider_config is a comma separated "key: value" pairs, e.g. "key1: value, key2: value2".
#Refer to https://docs.docker.com/registry/configuration/#storage for all available configuration.
registry_storage_provider_config =

3)Log files, you can get them by package the /var/log/harbor/ .

Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: Initializing database
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: 2018-01-31 20:45:18 139808786278336 [ERROR] InnoDB: preallocating 12582912 bytes for file ./ibdata1 failed with error 22
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: 2018-01-31 20:45:18 139808786278336 [ERROR] InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: 2018-01-31 20:45:18 139808786278336 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata
1 file before trying to start up again.
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: 2018-01-31 20:45:18 139808786278336 [ERROR] Plugin 'InnoDB' init function returned error.
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: 2018-01-31 20:45:18 139808786278336 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: 2018-01-31 20:45:18 139808786278336 [ERROR] Unknown/unsupported storage engine: InnoDB
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: 2018-01-31 20:45:18 139808786278336 [ERROR] Aborting
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: Installation of system tables failed!  Examine the logs in
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: /var/lib/mysql/ for more information.
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: The problem could be conflicting information in an external
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: my.cnf files. You can ignore these by doing:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:     shell> /usr/bin/mysql_install_db --defaults-file=~/.my.cnf
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: You can also try to start the mysqld daemon with:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:     shell> /usr/sbin/mysqld --skip-grant --general-log &
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: and use the command line tool /usr/bin/mysql
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: to connect to the mysql database and look at the grant tables:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:     shell> /usr/bin/mysql -u root mysql
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:     mysql> show tables;
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: Try 'mysqld --help' if you have problems with paths.  Using
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: --general-log gives you a log in /var/lib/mysql/ that may be helpful.
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: The latest information about mysql_install_db is available at
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: MariaDB is hosted on launchpad; You can find the latest source and
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: email lists at http://launchpad.net/maria
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: Please check all of the above before submitting a bug report
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]: at http://mariadb.org/jira
Jan 31 21:45:18 172.20.0.1 docker/mysql[23304]:
Jan 31 21:45:19 172.20.0.1 docker/mysql[23304]: /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-updatedb.d/upgrade.sh
Jan 31 21:45:19 172.20.0.1 docker/mysql[23304]: Maria DB flag not found, the DB was created in mysql image, running upgrade...
Jan 31 21:45:19 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:20 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:21 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:22 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:23 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:24 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:25 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:26 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:27 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:28 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:29 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:30 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:31 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:32 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:33 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:34 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:35 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:36 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:37 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:38 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:39 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:40 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:41 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:42 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:43 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:44 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:46 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:47 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:48 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:49 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:50 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:51 172.20.0.1 docker/mysql[23304]: MySQL failed to start.
Jan 31 21:45:51 172.20.0.1 docker/mysql[23304]: /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-updatedb.d/upgrade.sh
Jan 31 21:45:51 172.20.0.1 docker/mysql[23304]: Maria DB flag not found, the DB was created in mysql image, running upgrade...
Jan 31 21:45:51 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:53 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:54 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:55 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:56 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:57 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:58 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:45:59 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:00 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:01 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:02 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:03 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:04 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:05 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:06 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:07 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:08 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:09 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:10 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:11 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:12 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:13 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:14 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:15 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:16 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:17 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:18 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:19 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...
Jan 31 21:46:20 172.20.0.1 docker/mysql[23304]: Waiting for MySQL start...

[bergtwvd]

I wonder if the issue is related to https://jira.mariadb.org/browse/MDEV-14244.

My filesystem type is ext3, Ubuntu 14.04 LTS.

What MariaDB version is used in Harbor?

[bergtwvd]

Version appears:

bergtwvd@app-docker99:~/harbor.1.3.0$ docker run -it vmware/harbor-db:v1.3.0 --version
mysqld  Ver 10.2.10-MariaDB for Linux on x86_64 (Source distribution)

Looks likes MariaDB needs to be updated to resolve the problem ....

[ywk253100]

Did you upgrade to 1.3.0 and 1.4.0-rc1 from an old version or did a new installation without any existing data?

[bergtwvd]

I installed from scratch, ie no existing data.

If you do a literal search on the error message you will find a number of hits, several of them pointing to the same cause.

PS.I temporarily installed v1.2.0 with mysql, and that works fine.

[bergtwvd]

Is more information needed? I hope you can include the MariaDB upgrade in the next release candidate, so that I can use the latest version. I do not want to be stuck at 1.2.0 with mysql server.

[reasonerjt]

The file './ibdata1' resides in the /data directory on the docker host.

[reasonerjt]

@bergtwvd we can reproduce this issue on ext3 fs, however, the MariaDB is pulled from an internal repo, so we'll need to wait for upstream to fix this issue, for 1.4.0 we may add a release note, and fix this in 1.4.1

Thanks for raising the issue!

[bergtwvd]

Thanks for looking into this. Obviously I hope a fix can make it into 1.4.0, but I understand this may be not feasable.

[seanandmengjia]

Is there another solution? Can i build the docker harbor-db(v1.4.0) myself?

[bergtwvd]

I just wonder if/when this issue will be resolved. At the moment I am stuck with an old version of Harbor that I cannot upgrade to the latest version due to this issue. Any forecast?

[reasonerjt]

Unfortunately the MariaDB in our internal software has not been updated yet, a ticket has been opened but I cannot give an accurate forecast.

[bergtwvd]

This is quite problematic since I do not control the host OS installation and the type of filesystem used. The IT department creates VMs from a standard image. The filesystem is ext3 by default.

[bergtwvd]

A small update from my end:
The problem exists for ext2 and ext3.
The problem is solved for ext4.

I have moved to ext4 in the meantime and run Harbor 1.4.0.

[reasonerjt]

Verified in latest rc that mariaDB has been upgraded to 10.2.14, closing the issue.