Improve the efficiency of BufferedBlockDevice #8703

davidsaada · 2018-11-10T21:41:36Z

Description

Previous implementation of BufferedBlockDevice was extremely naive: It had a cache used for both writing and reading, meaning that a read action would always flush the write cache as well. In addition, write operations were split to chunks of program units, meaning that each BufferedBlockDevice program call would trigger plenty of the underlying BD's calls of this API. Reads did pretty much the same.
This change does the following:

Cache is used only for write operations. Reads use a separate buffer helping against unaligned addresses/sizes (but its data doesn't live across operations).
Reads and writes are only split to a minimal number of chunks according to need.

Pull request type

[x] Fix
[ ] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[ ] Test update
[ ] Breaking change

davidsaada · 2018-11-11T10:20:16Z

@cmonr @0xc0170 Jenkins fails due to unavailability of the K66F board in CI (restarted a few times).

0xc0170 · 2018-11-12T08:56:42Z

@cmonr @0xc0170 Jenkins fails due to unavailability of the K66F board in CI (restarted a few times).

👍 It was reported, and should be fixed now. We will restart failing jobs soon

0xc0170

No documentation changes?

davidsaada · 2018-11-12T13:38:50Z

No documentation changes?

No - no need to. Functionality is just the same, just more efficient.

cmonr · 2018-11-12T13:50:55Z

features/storage/blockdevice/BufferedBlockDevice.cpp

@@ -26,7 +26,8 @@ static inline uint32_t align_down(bd_size_t val, bd_size_t size)
 }

 BufferedBlockDevice::BufferedBlockDevice(BlockDevice *bd)
-    : _bd(bd), _bd_program_size(0), _curr_aligned_addr(0), _flushed(true), _cache(0), _init_ref_count(0), _is_initialized(false)
+    : _bd(bd), _bd_program_size(0), _bd_read_size(0), _write_cache_addr(0), _write_cache_valid(false),


Does this constructor change not need new documentation?

It doesn't look like the constructor params changed. These are member initializers, which are internal.

Curse that lone :

0xc0170 · 2018-11-13T08:59:14Z

/morph build

mbed-ci · 2018-11-13T10:17:06Z

Build : SUCCESS

Build number : 3609
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/8703/

Triggering tests

/morph test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-11-13T12:37:53Z

Exporter Build : SUCCESS

Build number : 3213
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/8703/

mbed-ci · 2018-11-13T12:56:24Z

Test : FAILURE

Build number : 3385
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/8703/3385

davidsaada · 2018-11-13T13:32:09Z

@0xc0170 @cmonr Please don't run morph test yet. Out of all unrelated failures, one (on nrf51) seems to be related to the PR.

davidsaada · 2018-11-13T15:14:27Z

Failure in NRF51 fixed (all other failures are not related to this PR).

NirSonnenschein · 2018-11-14T08:55:58Z

/morph export-build

NirSonnenschein · 2018-11-14T09:01:47Z

/morph build

mbed-ci · 2018-11-14T11:11:01Z

Build : SUCCESS

Build number : 3618
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/8703/

Triggering tests

/morph test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-11-14T12:19:25Z

Exporter Build : SUCCESS

Build number : 3223
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/8703/

mbed-ci · 2018-11-14T15:58:01Z

Exporter Build : SUCCESS

Build number : 3226
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/8703/

mbed-ci · 2018-11-14T18:03:04Z

Test : FAILURE

Build number : 3401
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/8703/3401

cmonr · 2018-11-14T19:34:36Z

Test failures appear to be a whole bunch of sporadic netsocket echo issues, unrelated to CI.

Will restart when able.

geky

This looks great 👍

geky · 2018-11-14T22:09:22Z

features/storage/blockdevice/BufferedBlockDevice.cpp

@@ -110,36 +124,52 @@ int BufferedBlockDevice::sync()

 int BufferedBlockDevice::read(void *b, bd_addr_t addr, bd_size_t size)
 {
-    MBED_ASSERT(_cache);
+    MBED_ASSERT(_write_cache);


Should this be?

Suggested change

MBED_ASSERT(_write_cache);

MBED_ASSERT(_read_cache);

Also should there be a similar assert in BlockDevice::program?

@davidsaada ^^^

@cmonr, given the dependencies, it's probably better to just move forward with testing. The assert is minor and can be fixed after the release (or if @davidsaada has to make further changes).

@geky Alright, that sounds reasonable.

Used those damned netsocket test failures to fix these asserts.

0xc0170 · 2018-11-15T07:34:41Z

/morph test

cmonr · 2018-11-15T17:35:25Z

@davidsaada Was this PR just rebased?

davidsaada · 2018-11-15T17:54:29Z

Was this PR just rebased?

Rebased it as part of the fixes (assert fixes).

cmonr · 2018-11-15T22:32:39Z

Info: This PR has now been bundled into a rollup PR (#8760).

The CI failures that this PR saw do not appear to be the fault of the PR. To save time in retesting this PR, along with other PRs that experienced similar issues, it has been bundled into a rollup PR.

If any more commits are made in this PR, this PR will remain open and have to go through CI on its own.

cmonr · 2018-11-16T00:55:09Z

Info: This PR has been re-bundled into a new rollup PR (#8763).

The previous rollup found an issue with a bundled PR after new devices were added into master. The PR needing work has been removed.

No further work is needed here, as once that PR is merged, this PR will also be closed and marked as merged.
If any more commits are made in this PR, this PR will remain open and have to go through CI on its own.

davidsaada mentioned this pull request Nov 11, 2018

KVStore & derived classes: design docs, implementation & configuration #8667

Merged

0xc0170 added the needs: review label Nov 12, 2018

0xc0170 requested a review from a team November 12, 2018 08:56

0xc0170 approved these changes Nov 12, 2018

View reviewed changes

dannybenor approved these changes Nov 12, 2018

View reviewed changes

cmonr added needs: CI release-version: 5.10.4 and removed needs: review labels Nov 12, 2018

cmonr reviewed Nov 12, 2018

View reviewed changes

cmonr added release-version: 5.11.0-rc1 and removed release-version: 5.10.4 labels Nov 12, 2018

davidsaada force-pushed the david_buffered_bd_perf branch from 4cd7813 to 2629d2d Compare November 13, 2018 15:12

geky approved these changes Nov 14, 2018

View reviewed changes

theamirocohen mentioned this pull request Nov 15, 2018

Add block device test for small data sizes #8740

Merged

Improve the efficiency of BufferedBlockDevice

50836e7

davidsaada force-pushed the david_buffered_bd_perf branch from 2629d2d to 50836e7 Compare November 15, 2018 09:52

cmonr added the rollup PR: B label Nov 15, 2018

cmonr mentioned this pull request Nov 15, 2018

Rollup PR: Retest CI-troubled PRs targeted for 5.11 #8760

Closed

cmonr mentioned this pull request Nov 16, 2018

Rollup PR: Retest CI-troubled PRs target for 5.11 v2 #8763

Merged

0xc0170 merged commit 50836e7 into ARMmbed:master Nov 16, 2018

0xc0170 removed the needs: CI label Nov 16, 2018

davidsaada deleted the david_buffered_bd_perf branch December 6, 2018 13:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve the efficiency of BufferedBlockDevice #8703

Improve the efficiency of BufferedBlockDevice #8703

davidsaada commented Nov 10, 2018

davidsaada commented Nov 11, 2018

0xc0170 commented Nov 12, 2018

0xc0170 left a comment

davidsaada commented Nov 12, 2018

cmonr Nov 12, 2018

geky Nov 12, 2018

cmonr Nov 12, 2018

0xc0170 commented Nov 13, 2018

mbed-ci commented Nov 13, 2018

mbed-ci commented Nov 13, 2018

mbed-ci commented Nov 13, 2018

davidsaada commented Nov 13, 2018

davidsaada commented Nov 13, 2018

NirSonnenschein commented Nov 14, 2018 •

edited

Loading

NirSonnenschein commented Nov 14, 2018

mbed-ci commented Nov 14, 2018

mbed-ci commented Nov 14, 2018

mbed-ci commented Nov 14, 2018

mbed-ci commented Nov 14, 2018

cmonr commented Nov 14, 2018

geky left a comment

geky Nov 14, 2018

cmonr Nov 14, 2018

geky Nov 14, 2018

cmonr Nov 15, 2018

davidsaada Nov 15, 2018

0xc0170 commented Nov 15, 2018

cmonr commented Nov 15, 2018

davidsaada commented Nov 15, 2018

cmonr commented Nov 15, 2018

cmonr commented Nov 16, 2018

Improve the efficiency of BufferedBlockDevice #8703

Improve the efficiency of BufferedBlockDevice #8703

Conversation

davidsaada commented Nov 10, 2018

Description

Pull request type

davidsaada commented Nov 11, 2018

0xc0170 commented Nov 12, 2018

0xc0170 left a comment

Choose a reason for hiding this comment

davidsaada commented Nov 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xc0170 commented Nov 13, 2018

mbed-ci commented Nov 13, 2018

Triggering tests

mbed-ci commented Nov 13, 2018

mbed-ci commented Nov 13, 2018

davidsaada commented Nov 13, 2018

davidsaada commented Nov 13, 2018

NirSonnenschein commented Nov 14, 2018 • edited Loading

NirSonnenschein commented Nov 14, 2018

mbed-ci commented Nov 14, 2018

Triggering tests

mbed-ci commented Nov 14, 2018

mbed-ci commented Nov 14, 2018

mbed-ci commented Nov 14, 2018

cmonr commented Nov 14, 2018

geky left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xc0170 commented Nov 15, 2018

cmonr commented Nov 15, 2018

davidsaada commented Nov 15, 2018

cmonr commented Nov 15, 2018

cmonr commented Nov 16, 2018

NirSonnenschein commented Nov 14, 2018 •

edited

Loading