New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

qa/rgw: add multisite suite to configure and run multisite tests #14688

Merged
merged 14 commits into from May 19, 2017

Conversation

Projects
None yet
2 participants
@cbodley
Contributor

cbodley commented Apr 20, 2017

work in progress for a multisite test suite in teuthology

TODO:

  • create zone pools depending on rgw.ec_data_pool
  • apply rgw compression type setting for created zones
  • add rgw_multisite_tests task to run multisite tests using configuration from rgw-multisite task
  • add nose as dependency to teuthology (see ceph/teuthology#1069)
@cbodley

This comment has been minimized.

Contributor

cbodley commented May 2, 2017

three-zone tests are currently failing, testing #14915 as a potential fix

@cbodley cbodley changed the title from [DNM] qa/rgw: add separate rgw_multisite task for multisite configuration to [DNM] qa/rgw: add multisite suite to configure and run multisite tests May 5, 2017

@@ -221,6 +218,8 @@ def ship_apache_configs(ctx, config, role_endpoints, on_client = None,
finally:
log.info('Removing apache config...')
for client in clients_to_create_as:
cluster_name, daemon_type, client_id = teuthology.split_role(client)
client_with_cluster = '.'.join((cluster_name, daemon_type, client_id))

This comment has been minimized.

@oritwas

oritwas May 16, 2017

Contributor

can you add some logging?

This comment has been minimized.

@cbodley

cbodley May 16, 2017

Contributor

what would you like to see logged here? just that it's getting the right value for client_with_cluster? here's a snippet from a teuthology.log that shows this in action:

INFO:tasks.rgw.client.0:Stopped
INFO:teuthology.orchestra.run.mira114:Running: 'rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock'
INFO:tasks.rgw:Stopping apache...
INFO:tasks.rgw:Removing apache config...
INFO:teuthology.orchestra.run.mira114:Running: 'rm -f /home/ubuntu/cephtest/apache/apache.ceph.client.0.conf && rm -f /home/ubuntu/cephtest/apache/htdocs.ceph.client.0/rgw.fcgi'
INFO:tasks.rgw:Cleaning up apache directories...
INFO:teuthology.orchestra.run.mira114:Running: 'rm -rf /home/ubuntu/cephtest/apache/tmp.ceph.client.0 && rmdir /home/ubuntu/cephtest/apache/htdocs.ceph.client.0'
INFO:teuthology.orchestra.run.mira114:Running: 'rmdir /home/ubuntu/cephtest/apache'
DEBUG:teuthology.run_tasks:Unwinding manager ceph

This comment has been minimized.

@oritwas

oritwas May 17, 2017

Contributor

I wanted the cluster name and client id.
I see from the logs the client id is already logged and I can find out what is cluster name from the teuthology node.
No need to add more logging

# separate region info so only clients are keys in config
regions = config['regions']
del config['regions']
regions = config.pop('regions', {})

This comment has been minimized.

@oritwas

oritwas May 16, 2017

Contributor

maybe it is time to change it to zonegroups :)
It can be in a separate pr ...

This comment has been minimized.

@cbodley

cbodley May 16, 2017

Contributor

yeah, i have a https://github.com/cbodley/ceph/commits/wip-rgw-agent-cleanup branch that removes this stuff

@@ -97,6 +89,17 @@ def meta_sync_status(zone):
return period, realm_epoch, num_shards, markers
def meta_sync_status(zone):
for _ in range(60):

This comment has been minimized.

@oritwas

oritwas May 16, 2017

Contributor

can you add a comment why 60?
I see it used a few times maybe a const?

This comment has been minimized.

@cbodley

cbodley May 16, 2017

Contributor

yeah, i just settled on a total of 5 minutes. i'll make this configurable along with the sleep in set_master_zone()

def set_master_zone(zone):
zone.modify(zone.cluster, ['--master'])
zonegroup = zone.zonegroup
zonegroup.period.update(zone, commit=True)
zonegroup.master_zone = zone
# wait for reconfiguration, so that later metadata requests go to the new master
time.sleep(5)
time.sleep(15)

This comment has been minimized.

@oritwas

oritwas May 16, 2017

Contributor

maybe it should be configurable?

This comment has been minimized.

@cbodley

cbodley May 16, 2017

Contributor

good idea! then we can keep it at 5 for test_multi.py, and raise it as much as we need to for teuthology

cbodley added some commits Apr 12, 2017

qa/rgw: add symlink to qa/tasks/rgw_multi
Signed-off-by: Casey Bodley <cbodley@redhat.com>
qa/rgw: add rgw_multisite task based on rgw_multi
Signed-off-by: Casey Bodley <cbodley@redhat.com>
qa/rgw: move startup polling logic to util/rgw.py
Signed-off-by: Casey Bodley <cbodley@redhat.com>
qa/rgw: move compression type out of ceph.conf
this makes the 'compression type' setting global to all gateways, and
makes the setting visible to other tasks in ctx.rgw.compression_type

Signed-off-by: Casey Bodley <cbodley@redhat.com>
test/rgw: add import for StringIO
also removes unnecessary tuple parens

Signed-off-by: Casey Bodley <cbodley@redhat.com>
qa/rgw: add rgw_multisite_tests task to run tests
Signed-off-by: Casey Bodley <cbodley@redhat.com>
qa/rgw: fixes for cluster name on cleanup
Signed-off-by: Casey Bodley <cbodley@redhat.com>
qa/rgw: add multisite suite
Signed-off-by: Casey Bodley <cbodley@redhat.com>
test/rgw: add 5min timeouts to sync checkpoints
if multisite tests are going to run in teuthology, they can't loop forever

Signed-off-by: Casey Bodley <cbodley@redhat.com>
test/rgw: dont assume zone1 and zone2 are on different clusters
Signed-off-by: Casey Bodley <cbodley@redhat.com>
qa/rgw: add log_level argument to rgwadmin()
changes default level from info to debug

Signed-off-by: Casey Bodley <cbodley@redhat.com>
test/rgw: add timing configuration
Signed-off-by: Casey Bodley <cbodley@redhat.com>
test/rgw: add timing config options to test_multi.py
Signed-off-by: Casey Bodley <cbodley@redhat.com>
qa/rgw: add test config to rgw_multisite_tests task
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley

This comment has been minimized.

Contributor

cbodley commented May 18, 2017

the latest run of the full rgw suite with --subset=1/6 --filter frontend/civetweb.yaml was all green: http://pulpito.ceph.com/cbodley-2017-05-17_14:49:58-rgw-master---basic-smithi/

the rgw:multisite suite pretty much always fails with valgrind errors (and osd crashes on shutdown), but the multisite tests themselves are passing:
http://pulpito.ceph.com/cbodley-2017-05-16_10:46:23-rgw:multisite-master---basic-smithi/

INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
INFO:tasks.rgw_multisite_tests:Ran 11 tests in 921.748s
INFO:tasks.rgw_multisite_tests:
INFO:tasks.rgw_multisite_tests:OK

@cbodley cbodley changed the title from [DNM] qa/rgw: add multisite suite to configure and run multisite tests to qa/rgw: add multisite suite to configure and run multisite tests May 18, 2017

@cbodley cbodley merged commit cff53b2 into ceph:master May 19, 2017

3 checks passed

Signed-off-by all commits in this PR are signed
Details
Unmodifed Submodules submodules for project are unmodified
Details
default Build finished.
Details

@cbodley cbodley deleted the cbodley:wip-rgw-multi-suite branch May 19, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment