test fix for mysql start timeout [WIP] [DO NOT MERGE] #1214

mancdaz · 2016-07-01T13:56:11Z

Here I'm testing a fix in OSA, but changing the submodule to point at
my fork of OSA where I have made a patch. I have to do it this way
because OSA doesn't do upgrade testing of mariadb between kilo and
liberty, which explains why we're only seeing this mostly in rpco.

The fix is essentially to wait longer between mysql start attempts after doing a mariadb upgrade.

Please, feel free to recheck_upgrade on this because that's what I'm testing here.

FYI this is the OSA fix I am testing https://github.com/mancdaz/openstack-ansible/commits/mysql-start

Connects #1201

mancdaz · 2016-07-01T15:22:31Z

if this passes, please continue to recheck the upgrade job over and over again

Don't care about the ceph or swift jobs for this particular PR

mancdaz · 2016-07-05T19:36:49Z

back to square one:

17:02:01 TASK: [galera_server | Start MySQL] ******************************************* 
17:02:01 <172.29.239.164> ESTABLISH CONNECTION FOR USER: root
17:02:01 <172.29.239.164> REMOTE_MODULE service name=mysql state=started
17:02:01 <172.29.239.164> EXEC ssh -C -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=120 172.29.239.164 /bin/sh -c 'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'
17:22:01 Build timed out (after 20 minutes). Marking the build as aborted.
17:22:01 Build was aborted

http://jenkins.propter.net/job/RPC-AIO/8484/consoleFull

mancdaz · 2016-07-05T20:55:35Z

Seems like there's a sneaky start of mariadb 10 happening when the package is being installed, even though it's mean to be denied by the policy:

17:01:34 TASK: [galera_server | Install galera packages] *******************************
17:01:34 <172.29.239.164> ESTABLISH CONNECTION FOR USER: root
17:01:34 <172.29.239.164> REMOTE_MODULE apt pkg=mariadb-client,mariadb-galera-server-10.0,galera-3,qpress,rsync,socat state=latest
17:01:34 <172.29.239.164> EXEC ssh -C -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=120 172.29.239.164 /bin/sh -c 'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'
17:01:56 changed: [jrpcaioiad-dd9_galera_container-96956b05] => (item=mariadb-client,mariadb-galera-server-10.0,galera-3,qpress,rsync,socat) => {"attempts": 0, "changed": true, "item": "mariadb-client,mariadb-galera-server-10.0,galera-3,qpress,rsync,socat", "stderr": "All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of stop.

Configuration file '/etc/init.d/mysql'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
 ==> Keeping old config file as default.

Configuration file '/etc/mysql/conf.d/mysqld_safe_syslog.cnf'
 ==> Deleted (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
 ==> Keeping old config file as default.
All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of stop.
160705 17:01:45 [Note] /usr/sbin/mysqld (mysqld 10.0.26-MariaDB-1~trusty-wsrep) starting as process 18684 ...
All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of start.
", "stdout": "Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  mailx mariadb-galera-test tinyca
The following NEW packages will be installed:
  mariadb-galera-server-10.0 qpress
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.4 MB of archives.
After this operation, 98.3 MB of additional disk space will be used.
Get:1 https://repo.percona.com/apt/ trusty/main qpress amd64 11-1.trusty [26.4 kB]
Preconfiguring packages ...
Fetched 10.4 MB in 0s (11.2 MB/s)
Selecting previously unselected package mariadb-galera-server-10.0.
(Reading database ... 23708 files and directories currently installed.)
Preparing to unpack .../mariadb-galera-server-10.0_10.0.26+maria-1~trusty_amd64.deb ...
Unpacking mariadb-galera-server-10.0 (10.0.26+maria-1~trusty) ...
Selecting previously unselected package qpress.
Preparing to unpack .../qpress_11-1.trusty_amd64.deb ...
Unpacking qpress (11-1.trusty) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up mariadb-galera-server-10.0 (10.0.26+maria-1~trusty) ...
Installing new version of config file /etc/logrotate.d/mysql-server ...
Installing new version of config file /etc/mysql/conf.d/tokudb.cnf ...
Setting up qpress (11-1.trusty) ...
Processing triggers for ureadahead (0.100.0-16) ...

Specifically note this line:

160705 17:01:45 [Note] /usr/sbin/mysqld (mysqld 10.0.26-MariaDB-1~trusty-wsrep) starting as process 18684 ...

what are you doing there buddy?

mancdaz · 2016-07-05T21:43:56Z

using jenkins build 8491:

galera log: http://jenkins.propter.net/job/RPC-AIO/8491/artifact/archive/openstack/jrpcaioiad-dd9_galera_container-96956b05/galera_server_error.log

specific bit of the galera log we are interested in showing the multiple starts/stops of mysqld/innodb/wsrep: http://paste.openstack.org/show/526090/

ansible console log: http://jenkins.propter.net/job/RPC-AIO/8491/consoleFull

specific bit of ansible log showing the sneaky mysqld start at the package install time:

http://paste.openstack.org/show/526091/

git-harry · 2016-07-06T08:24:38Z

@mancdaz are you sure it's starting, the following two lines in the log suggest it was prevented

160705 17:01:45 [Note] /usr/sbin/mysqld (mysqld 10.0.26-MariaDB-1~trusty-wsrep) starting as process 18684 ...
All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of start.

mancdaz · 2016-07-06T09:20:14Z

@git-harry I'm not sure of anything tbh, just continuing to pour through the logfiles and try and note anything odd.

However, the timing of that line does suggest that something is trying to start up at the point of install, and indeed the mysql error log shows lots of activity around then. If it was truly being denied by policy-rc.d, I would not expect it to be assigned a PID.

mancdaz · 2016-07-06T09:23:04Z

More notes:

In a successful startup, the following is in the mysql log:

160704 21:37:03 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
160704 21:37:03 [Note] WSREP: restore pc from disk failed

In a failed startup:

[Note] WSREP: restore pc from disk successfully

The existence of a gvwstate.dat file is evidence of a non-graceful shutdown (i.e. crash) prior to this startup attempt. So we actually want it to not find that file.

May mean nothing, who knows

mancdaz · 2016-07-06T10:43:36Z

So, adding a task to pgrep for mysqld and look at netstat right after the mariadb-server package is installed (bearing in mind that policy-rc.d should not allow it to be running), shows that it is indeed running (or at least the galera wsrep part of it is listening on 4567):

10:15:54 TASK: [galera_server | pgrep mysqld] ****************************************** 
10:15:54 <172.29.237.238> ESTABLISH CONNECTION FOR USER: root
10:15:54 <172.29.237.238> REMOTE_MODULE command pgrep mysqld
10:15:54 <172.29.237.238> EXEC ssh -C -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=120 172.29.237.238 /bin/sh -c 'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'
10:15:54 changed: [jrpcaioiad-831_galera_container-6d11afca] => {"changed": true, "cmd": ["pgrep", "mysqld"], "delta": "0:00:00.004601", "end": "2016-07-06 10:15:54.512153", "rc": 0, "start": "2016-07-06 10:15:54.507552", "stderr": "", "stdout": "575\n1480", "warnings": []}
10:15:54 
10:15:54 TASK: [galera_server | netstat] *********************************************** 
10:15:54 <172.29.237.238> ESTABLISH CONNECTION FOR USER: root
10:15:54 <172.29.237.238> REMOTE_MODULE command netstat -pntl
10:15:54 <172.29.237.238> EXEC ssh -C -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=120 172.29.237.238 /bin/sh -c 'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'
10:15:54 changed: [jrpcaioiad-831_galera_container-6d11afca] => {"changed": true, "cmd": ["netstat", "-pntl"], "delta": "0:00:00.007537", "end": "2016-07-06 10:15:54.596204", "rc": 0, "start": "2016-07-06 10:15:54.588667", "stderr": "", "stdout": "Active Internet connections (only servers)\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name\ntcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      528/sshd        \ntcp        0      0 0.0.0.0:4567            0.0.0.0:*               LISTEN      1480/mysqld     \ntcp6       0      0 :::22                   :::*                    LISTEN      528/sshd        ", "warnings": []}

Pulling out the important bits:

netstat:

command: ["netstat", "-pntl"]
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      528/sshd
tcp        0      0 0.0.0.0:4567            0.0.0.0:*               LISTEN      1480/mysqld
tcp6       0      0 :::22                   :::*                    LISTEN      528/sshd        "

pgrep:

command: ["pgrep", "mysqld"]
"575
1480"

mancdaz · 2016-07-06T10:52:19Z

In an upgrade job, nodes 1,2 of the cluster are restarted at the beginning of the liberty upgrade script, and then node 0 is restarted. In a job where the later mariadb upgrade fails, it's useful to note that earlier when those restarts happened, the cluster was left in a bad state:

160705 16:38:22 [Warning] WSREP: No re-merged primary component found.
160705 16:38:22 [Warning] WSREP: No bootstrapped primary component found.
160705 16:38:22 [ERROR] WSREP: gcs/src/gcs_state_msg.cpp:gcs_state_msg_get_quorum():794: Failed to establish quorum.
160705 16:38:22 [Note] WSREP: Quorum results:
    version    = 3,
    component  = NON-PRIMARY,
    conf_id    = -1,
    members    = 0/3 (joined/total),
    act_id     = -1,
    last_appl. = -1,
    protocols  = -1/-1/-1 (gcs/repl/appl),
    group UUID = 00000000-0000-0000-0000-000000000000
160705 16:38:22 [Note] WSREP: Flow-control interval: [28, 28]
160705 16:38:22 [Note] WSREP: Received NON-PRIMARY.
160705 16:38:22 [Note] WSREP: New cluster view: global state: e5c8e624-42c4-11e6-b8fa-13e99514039f:8970, view# -1: non-Primary, number of nodes: 3, my index: 2, protocol version -1
160705 16:38:22 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160705 16:38:24 [Note] WSREP: (e5c841c6, 'tcp://0.0.0.0:4567') turning message relay requesting off

In a 'good' job, the cluster gets itself back into a good state after those container restarts:

160704 21:12:10 [Warning] WSREP: No re-merged primary component found.
160704 21:12:10 [Note] WSREP: Bootstrapped primary 00000000-0000-0000-0000-000000000000 found: 3.
160704 21:12:10 [Note] WSREP: Quorum results:
    version    = 3,
    component  = PRIMARY,
    conf_id    = -1,
    members    = 3/3 (joined/total),
    act_id     = 9075,
    last_appl. = -1,
    protocols  = 0/7/3 (gcs/repl/appl),
    group UUID = acf766fa-4221-11e6-92bc-d7b4acbf1b0e
160704 21:12:10 [Note] WSREP: Flow-control interval: [28, 28]
160704 21:12:10 [Note] WSREP: Restored state OPEN -> JOINED (9075)
160704 21:12:10 [Note] WSREP: New cluster view: global state: acf766fa-4221-11e6-92bc-d7b4acbf1b0e:9075, view# 0: Primary, number of nodes: 3, my index: 1, protocol version 3
160704 21:12:10 [Note] WSREP: SST complete, seqno: 9075
160704 21:12:10 [Note] WSREP: Member 1.0 (jrpcaioiad-265_galera_container-bf6e1285) synced with group.
160704 21:12:10 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 9075)
160704 21:12:10 [Note] WSREP: Member 2.0 (jrpcaioiad-265_galera_container-9c839293) synced with group.
160704 21:12:10 [Note] WSREP: Member 0.0 (jrpcaioiad-265_galera_container-25d21a22) synced with group.
160704 21:12:10 InnoDB: The InnoDB memory heap is disabled
160704 21:12:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160704 21:12:10 InnoDB: Compressed tables use zlib 1.2.8
160704 21:12:10 InnoDB: Using Linux native AIO
160704 21:12:10 InnoDB: Initializing buffer pool, size = 512.0M
160704 21:12:10 InnoDB: Completed initialization of buffer pool
160704 21:12:10 InnoDB: highest supported file format is Barracuda.
160704 21:12:11  InnoDB: Waiting for the background threads to start
160704 21:12:12 Percona XtraDB (http://www.percona.com) 5.5.49-MariaDB-37.9 started; log sequence number 23397856
160704 21:12:12 [Note] Plugin 'FEEDBACK' is disabled.
160704 21:12:12 [Note] Server socket created on IP: '0.0.0.0'.
160704 21:12:12 [Note] Event Scheduler: Loaded 0 events
160704 21:12:12 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.50-MariaDB-1~trusty-wsrep'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution, wsrep_25.14.r9949137
160704 21:12:12 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160704 21:12:12 [Note] WSREP: REPL Protocols: 7 (3, 2)
160704 21:12:12 [Note] WSREP: Service thread queue flushed.
160704 21:12:12 [Note] WSREP: Assign initial position for certification: 9075, protocol version: 3
160704 21:12:12 [Note] WSREP: Service thread queue flushed.
160704 21:12:12 [Note] WSREP: Synchronized with group, ready for connections
160704 21:12:12 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160704 21:12:13 [Note] WSREP: (acf68560, 'tcp://0.0.0.0:4567') turning message relay requesting off

mancdaz · 2016-07-06T20:51:28Z

OK so it's not mariadb 10 that's starting right on install, but an old lingering mariadb 5.5 daemon that never got shut down properly from earlier. We can see this by looking at the PID we saw earlier:

netstat:

command: ["netstat", "-pntl"]
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      528/sshd
tcp        0      0 0.0.0.0:4567            0.0.0.0:*               LISTEN      1480/mysqld
tcp6       0      0 :::22                   :::*                    LISTEN      528/sshd        "

And looking in the mariadb log for that container:

160706  9:52:02 [Note] /usr/sbin/mysqld (mysqld 5.5.50-MariaDB-1~trusty-wsrep) starting as process 1480

Shortly after this, the cluster fails to establish quorum and goes into error state:

160706  9:52:04 [Warning] WSREP: No bootstrapped primary component found.
160706  9:52:04 [ERROR] WSREP: gcs/src/gcs_state_msg.cpp:gcs_state_msg_get_quorum():794: Failed to establish quorum.
160706  9:52:04 [Note] WSREP: Quorum results:
    version    = 3,
    component  = NON-PRIMARY,
    conf_id    = -1,
    members    = 0/3 (joined/total),
    act_id     = -1,
    last_appl. = -1,
    protocols  = -1/-1/-1 (gcs/repl/appl),
    group UUID = 00000000-0000-0000-0000-000000000000
160706  9:52:04 [Note] WSREP: Flow-control interval: [28, 28]
160706  9:52:04 [Note] WSREP: Received NON-PRIMARY.
160706  9:52:04 [Note] WSREP: New cluster view: global state: c77fc194-4354-11e6-8737-6215be425fe5:9163, view# -1: non-Primary, number of nodes: 3, my index: 1, protocol version -1
160706  9:52:04 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160706  9:52:05 [Note] WSREP: (c77ed8c5, 'tcp://0.0.0.0:4567') turning message relay requesting off

For whatever reason, this means that the mariadb 5.5 daemon fails to shut down properly when we come to install/start mariadb 10.

Need to understand what is causing the cluster to go into error (fail to reach quorum) around the time of the container restarts at the beginning of the upgrade.

mancdaz · 2016-07-18T10:33:43Z

Current working theory is to gracefully stop mariadb on nodes prior to container restarts, to ensure data consistency and prevent nodes seeing themselves as 'crashed'.

Interestingly, stopping (gracefully or otherwise) nodes 2/3 simultaneously causes the manifestation of this galera bug: https://bugs.launchpad.net/galera/+bug/1217225

Now trying stopping mariadb in serial to see if we get what we want

mancdaz · 2016-07-18T16:11:19Z

recording upgrade tests here:

overall fail, since the patch was supposed to remove all occurrences of mysql start fails.

mancdaz · 2016-07-20T11:11:22Z

turns out my graceful stop of mariadb on the master node was being skipped because of https://bugs.launchpad.net/openstack-ansible/+bug/1604775

trying again now with a workaround

mancdaz · 2016-07-20T12:16:21Z

The story deepens. Turns out that the part of the lxc_container_create role that adds some autodev stuff for containers in liberty, thusly:

https://github.com/openstack/openstack-ansible/blob/liberty/playbooks/roles/lxc_container_create/tasks/container_create.yml#L46-L55

causes a secret container restart, so even though my patch is shutting mysql down prior to the later explicit container restart, there is a secret container restart which causes mysql to start again. Then, when it comes to the later explicit container restart, we're basically just back to forcefully crashing the mysql daemon without a graceful stop. le sigh.

mancdaz · 2016-07-20T13:55:01Z

w00t https://review.openstack.org/#/c/344834/ fixes the 'secret container restart' problem, and gives us more flexibility and control over when containers restart.

Testing again including that fix

mancdaz · 2016-07-20T14:02:30Z

mancdaz · 2016-07-21T12:44:02Z

Now testing a fully managed rolling upgrade/restart on the galera cluster

mancdaz · 2016-07-26T10:49:32Z

http://jenkins.propter.net/job/RPC-AIO/9842/consoleFull
http://jenkins.propter.net/job/RPC-AIO/9845/consoleFull
http://jenkins.propter.net/job/RPC-AIO/9846/consoleFull
http://jenkins.propter.net/job/RPC-AIO/9847/consoleFull
http://jenkins.propter.net/job/RPC-AIO/9848/consoleFull

All failed in kilo when deploying maas:

Task Failed: Install MaaS / rpc_maas / Zero entities with label {{ inventory_hostname }}{{ maas_fqdn_extension }} exist

mancdaz · 2016-07-26T12:17:42Z

http://jenkins.propter.net/job/RPC-AIO/9855/consoleFull
http://jenkins.propter.net/job/RPC-AIO/9856/consoleFull
http://jenkins.propter.net/job/RPC-AIO/9857/consoleFull
http://jenkins.propter.net/job/RPC-AIO/9858/consoleFull
http://jenkins.propter.net/job/RPC-AIO/9859/consoleFull

mancdaz · 2016-07-26T13:36:29Z

all jenkins jobs currently failing due to maas rate limiting

mancdaz · 2016-07-26T15:45:56Z

http://jenkins.propter.net/job/RPC-AIO/9924/consoleFull: install elasticsearch <2.1.0 fail (liberty)
http://jenkins.propter.net/job/RPC-AIO/9950/consoleFull: install elasticsearch <2.1.0 fail (liberty)

I appear to have introduced some kind of problem with elasticsearch installs, maybe an artifact of the way I was rebuilding the same test, or some kind of problem with a lack of rebase for a while. Hoping that a rebase will sort that issue.

The tests are at least passing the mysql upgrade part!

Tests with reverted jimmy patch to try and fix the elsaticsearch constraint:
http://jenkins.propter.net/job/RPC-AIO/10047/consoleFull:

straight liberty build to test jimmy patch revert
http://jenkins.propter.net/job/RPC-AIO/10048/consoleFull
http://jenkins.propter.net/job/RPC-AIO/10049/consoleFull

pointing at my own fork of openstack-ansible since that is where the actual fix is.

mancdaz · 2016-07-28T18:11:11Z

Ok so aside from issues with global requirements , we have 11 successful tests of the mysql changes.

Going to do a few more for fun

mancdaz · 2016-08-01T08:40:27Z

So, the story comes to an end. The patch was submitted in OSA https://review.openstack.org/#/c/347195/ and has merged. Yay for better control of mariadb/galera in upgrades.

mancdaz added the status-dont-merge label Jul 1, 2016

mancdaz force-pushed the issues/1201/liberty-12.2 branch 5 times, most recently from 0699087 to 79cdcb6 Compare July 4, 2016 19:39

mancdaz force-pushed the issues/1201/liberty-12.2 branch 2 times, most recently from 0c4473f to 71203fe Compare July 6, 2016 08:16

mancdaz force-pushed the issues/1201/liberty-12.2 branch 4 times, most recently from fd131ac to 8f3e9f8 Compare July 14, 2016 12:39

mancdaz force-pushed the issues/1201/liberty-12.2 branch 2 times, most recently from deda927 to d2f4e8e Compare July 18, 2016 10:32

mancdaz force-pushed the issues/1201/liberty-12.2 branch 5 times, most recently from f82945e to 040e01b Compare July 18, 2016 12:42

mancdaz force-pushed the issues/1201/liberty-12.2 branch from 040e01b to 70910fe Compare July 18, 2016 15:50

mancdaz force-pushed the issues/1201/liberty-12.2 branch 2 times, most recently from 722b31b to 2316e67 Compare July 20, 2016 11:07

mancdaz force-pushed the issues/1201/liberty-12.2 branch from 2316e67 to 0ee4df7 Compare July 20, 2016 13:53

mancdaz force-pushed the issues/1201/liberty-12.2 branch from 0ee4df7 to 2984d9f Compare July 21, 2016 12:33

mancdaz force-pushed the issues/1201/liberty-12.2 branch from 2984d9f to 8bce6b5 Compare July 21, 2016 21:51

b3rn4rd0s added the feature-gating label Jul 25, 2016

mancdaz force-pushed the issues/1201/liberty-12.2 branch from 8bce6b5 to 9eb1c2e Compare July 26, 2016 10:00

git-harry removed the feature-gating label Jul 27, 2016

mancdaz force-pushed the issues/1201/liberty-12.2 branch 2 times, most recently from 7d226c5 to 9eb1c2e Compare July 28, 2016 08:37

test fix for mysql-start problems

61a7f6f

pointing at my own fork of openstack-ansible since that is where the actual fix is.

mancdaz force-pushed the issues/1201/liberty-12.2 branch from 9eb1c2e to 61a7f6f Compare July 28, 2016 13:06

mancdaz closed this Aug 1, 2016

git-harry deleted the issues/1201/liberty-12.2 branch August 1, 2016 09:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test fix for mysql start timeout [WIP] [DO NOT MERGE] #1214

test fix for mysql start timeout [WIP] [DO NOT MERGE] #1214

mancdaz commented Jul 1, 2016 •

edited by b3rn4rd0s

mancdaz commented Jul 1, 2016 •

edited

mancdaz commented Jul 5, 2016

mancdaz commented Jul 5, 2016 •

edited

mancdaz commented Jul 5, 2016 •

edited

git-harry commented Jul 6, 2016

mancdaz commented Jul 6, 2016

mancdaz commented Jul 6, 2016

mancdaz commented Jul 6, 2016

mancdaz commented Jul 6, 2016 •

edited

mancdaz commented Jul 6, 2016

mancdaz commented Jul 18, 2016

mancdaz commented Jul 18, 2016 •

edited

mancdaz commented Jul 20, 2016

mancdaz commented Jul 20, 2016

mancdaz commented Jul 20, 2016

mancdaz commented Jul 20, 2016 •

edited

mancdaz commented Jul 21, 2016 •

edited

mancdaz commented Jul 26, 2016 •

edited

mancdaz commented Jul 26, 2016 •

edited

mancdaz commented Jul 26, 2016

mancdaz commented Jul 26, 2016 •

edited

mancdaz commented Jul 28, 2016

mancdaz commented Aug 1, 2016

test fix for mysql start timeout [WIP] [DO NOT MERGE] #1214

test fix for mysql start timeout [WIP] [DO NOT MERGE] #1214

Conversation

mancdaz commented Jul 1, 2016 • edited by b3rn4rd0s

mancdaz commented Jul 1, 2016 • edited

mancdaz commented Jul 5, 2016

mancdaz commented Jul 5, 2016 • edited

mancdaz commented Jul 5, 2016 • edited

git-harry commented Jul 6, 2016

mancdaz commented Jul 6, 2016

mancdaz commented Jul 6, 2016

mancdaz commented Jul 6, 2016

mancdaz commented Jul 6, 2016 • edited

mancdaz commented Jul 6, 2016

mancdaz commented Jul 18, 2016

mancdaz commented Jul 18, 2016 • edited

mancdaz commented Jul 20, 2016

mancdaz commented Jul 20, 2016

mancdaz commented Jul 20, 2016

mancdaz commented Jul 20, 2016 • edited

mancdaz commented Jul 21, 2016 • edited

mancdaz commented Jul 26, 2016 • edited

mancdaz commented Jul 26, 2016 • edited

mancdaz commented Jul 26, 2016

mancdaz commented Jul 26, 2016 • edited

mancdaz commented Jul 28, 2016

mancdaz commented Aug 1, 2016

mancdaz commented Jul 1, 2016 •

edited by b3rn4rd0s

mancdaz commented Jul 1, 2016 •

edited

mancdaz commented Jul 5, 2016 •

edited

mancdaz commented Jul 5, 2016 •

edited

mancdaz commented Jul 6, 2016 •

edited

mancdaz commented Jul 18, 2016 •

edited

mancdaz commented Jul 20, 2016 •

edited

mancdaz commented Jul 21, 2016 •

edited

mancdaz commented Jul 26, 2016 •

edited

mancdaz commented Jul 26, 2016 •

edited

mancdaz commented Jul 26, 2016 •

edited