Cache on the barrelhead #19581

adamemerson · 2017-12-18T19:15:35Z

RGW Cache improvements

cbodley · 2017-12-18T21:19:12Z

src/rgw/rgw_user.cc

@@ -297,7 +297,7 @@ int rgw_get_user_info_from_index(RGWRados * const store,
  list<rgw_cache_entry_info *> cache_info_entries;
  cache_info_entries.push_back(&cache_info);

-  uinfo_cache.put(store, key, &e, cache_info_entries);
+  uinfo_cache.put(store, key, &e, { &cache_info });


it looks like the cache_info_entries list above is no longer used

mattbenjamin · 2017-12-19T22:16:02Z

@adamemerson promising

adamemerson · 2017-12-21T17:34:55Z

Jenkins, retest this please.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Do not use std::list for the LRU. And really don't cons up a std::list just to pass a variable number of arguments to a function. (Use initializer_list instead.) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

This should get us better look up speeds. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Rename it to rgw_cache_expiry_interval, to be specific. It already controls the user cache, and will expire objects in the cache onto which those two are chained. Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

We had it in the chained caches, but it doesn't do much good if they just fetch objects out of the object cache. Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Now when we force a refetch of bucket info it will actually go to the OSD rather than simply using the objects in the object cache. Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mattbenjamin · 2017-12-21T21:11:38Z

http://pulpito.ceph.com/mbenjamin-2017-12-21_15:24:10-rgw-wip-rgw-22517---basic-smithi/

mattbenjamin · 2017-12-21T21:57:00Z

2017-12-21T21:49:25.384 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok
2017-12-21T21:49:25.385 INFO:tasks.rgw_multisite_tests:
2017-12-21T21:49:25.385 INFO:tasks.rgw_multisite_tests:======================================================================
2017-12-21T21:49:25.385 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_disable_enable
2017-12-21T21:49:25.385 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2017-12-21T21:49:25.385 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2017-12-21T21:49:25.385 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
2017-12-21T21:49:25.385 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2017-12-21T21:49:25.385 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-rgw-22517/qa/tasks/rgw_multi/tests.py", line 1004, in test_bucket_sync_disable_enable
2017-12-21T21:49:25.385 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket_name)
2017-12-21T21:49:25.385 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-rgw-22517/qa/tasks/rgw_multi/tests.py", line 398, in zonegroup_bucket_checkpoint
2017-12-21T21:49:25.385 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name)
2017-12-21T21:49:25.386 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-rgw-22517/qa/tasks/rgw_multi/tests.py", line 391, in zone_bucket_checkpoint
2017-12-21T21:49:25.386 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name)
2017-12-21T21:49:25.386 INFO:tasks.rgw_multisite_tests:AssertionError: finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=wyhalv-53
2017-12-21T21:49:25.386 INFO:tasks.rgw_multisite_tests:
2017-12-21T21:49:25.386 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2017-12-21T21:49:25.386 INFO:tasks.rgw_multisite_tests:Ran 21 tests in 2924.258s
2017-12-21T21:49:25.386 INFO:tasks.rgw_multisite_tests:
2017-12-21T21:49:25.386 INFO:tasks.rgw_multisite_tests:FAILED (failures=1)
2017-12-21T21:49:25.386 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
manager.enter()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/init.py", line 123, in enter
self.begin()
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-rgw-22517/qa/tasks/rgw_multisite_tests.py", line 68, in begin
raise RuntimeError('rgw multisite test failures')
RuntimeError: rgw multisite test failures

mattbenjamin · 2017-12-21T21:58:03Z

@cbodley @oritwas could one of you take a look at the multisite failure (bucket sync enable-disable)--should it block this PR?

cbodley · 2018-01-02T15:49:56Z

@cbodley @oritwas could one of you take a look at the multisite failure (bucket sync enable-disable)--should it block this PR?

does not look like a blocker. that same test case passed in the other multisite job

mattbenjamin

lgtm sir

mattbenjamin · 2018-01-02T16:00:56Z

@adamemerson I am getting a merge failure, can you please rebase?

mattbenjamin · 2018-01-02T16:01:20Z

@adamemerson never mind...

cbodley · 2018-01-03T19:06:42Z

@adamemerson now that this merged, i'm seeing some crashes in ObjectCache::touch_lru() when it calls lru.erase(lru_iter);. my best guess is that we're running afoul of the iterator invalidation rules (which are different between list and deque) when storing std::deque<string>::iterator lru_iter in ObjectCacheEntry

cbodley · 2018-01-03T19:13:55Z

added http://tracker.ceph.com/issues/22560 with stack trace

cbodley · 2018-01-03T21:30:36Z

src/rgw/rgw_cache.cc

@@ -135,8 +134,6 @@ void ObjectCache::put(string& name, ObjectCacheInfo& info, rgw_cache_entry_info
  ObjectCacheEntry& entry = iter->second;
  ObjectCacheInfo& target = entry.info;

-  invalidate_lru(entry);
-


separately, it looks like the removal of this call is causing some failures in s3tests - added back in #19768

adamemerson added DNM performance rgw labels Dec 18, 2017

cbodley reviewed Dec 18, 2017

View reviewed changes

adamemerson force-pushed the wip-cache-on-the-barrelhead branch 2 times, most recently from 84906b8 to b3249c3 Compare December 19, 2017 17:57

adamemerson force-pushed the wip-cache-on-the-barrelhead branch from a1a2c12 to 0ca74ce Compare December 20, 2017 22:09

adamemerson added needs-qa bug-fix and removed DNM labels Dec 21, 2017

adamemerson added 6 commits December 21, 2017 15:44

rgw: Have ChainedCacheImpl::find return an optional

d622aa0

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Cache data structure cleanup

afa51de

Do not use std::list for the LRU. And really don't cons up a std::list just to pass a variable number of arguments to a function. (Use initializer_list instead.) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Use unordered_map for cache_map

b48753b

This should get us better look up speeds. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add expiration in the object cache

82a7e6c

We had it in the chained caches, but it doesn't do much good if they just fetch objects out of the object cache. Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Plumb refresh logic into object cache

d997f65

Now when we force a refetch of bucket info it will actually go to the OSD rather than simply using the objects in the object cache. Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

adamemerson force-pushed the wip-cache-on-the-barrelhead branch from 0ca74ce to d997f65 Compare December 21, 2017 20:47

mattbenjamin added the wip-matt-testing label Dec 21, 2017

mattbenjamin approved these changes Jan 2, 2018

View reviewed changes

mattbenjamin merged commit 05bcf0c into ceph:master Jan 2, 2018

mattbenjamin mentioned this pull request Jan 3, 2018

jewel: rgw: resolve Random 500 errors in Swift PutObject (22517) #19769

Merged

cbodley reviewed Jan 3, 2018

View reviewed changes

cbodley mentioned this pull request Jan 5, 2018

luminous: rgw: Random 500 errors in Swift PutObject (needs cache fixes) #19788

Merged

adamemerson deleted the wip-cache-on-the-barrelhead branch January 29, 2018 19:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cache on the barrelhead #19581

Cache on the barrelhead #19581

adamemerson commented Dec 18, 2017

cbodley Dec 18, 2017

mattbenjamin commented Dec 19, 2017

adamemerson commented Dec 21, 2017

mattbenjamin commented Dec 21, 2017

mattbenjamin commented Dec 21, 2017

mattbenjamin commented Dec 21, 2017

cbodley commented Jan 2, 2018

mattbenjamin left a comment

mattbenjamin commented Jan 2, 2018

mattbenjamin commented Jan 2, 2018

cbodley commented Jan 3, 2018

cbodley commented Jan 3, 2018

cbodley Jan 3, 2018

Cache on the barrelhead #19581

Cache on the barrelhead #19581

Conversation

adamemerson commented Dec 18, 2017

cbodley Dec 18, 2017

Choose a reason for hiding this comment

mattbenjamin commented Dec 19, 2017

adamemerson commented Dec 21, 2017

mattbenjamin commented Dec 21, 2017

mattbenjamin commented Dec 21, 2017

mattbenjamin commented Dec 21, 2017

cbodley commented Jan 2, 2018

mattbenjamin left a comment

Choose a reason for hiding this comment

mattbenjamin commented Jan 2, 2018

mattbenjamin commented Jan 2, 2018

cbodley commented Jan 3, 2018

cbodley commented Jan 3, 2018

cbodley Jan 3, 2018

Choose a reason for hiding this comment