FlashIAP driver modifications #6140

davidsaada · 2018-02-20T12:58:15Z

Description

A few FlashIAP driver modifications

Support programming across sectors.
Support program size not aligned to page size.

Status

READY

0xc0170 · 2018-02-20T13:16:13Z

Make class singleton.
Support programming across sectors.
Support program size not aligned to page size.

It would be good if we split new API from bugfixing current implementation.
What I see here: add singleton would be a new commit (preferably separate PR) as this can't be added to patch releases.
The other two aim to improve the current implementation thus should be separated.

This splitting adds a bit of overhead (plus separate PR for new API and bugfixing) but brings more benefit in the long run (maintenance, review, etc).

I would suggest this PR become bugfixing (improving the current implementation), and singleton addition new PR.

This is stated in our workflow document:

Pull requests should fix a bug, add a feature or refactor.

davidsaada · 2018-02-20T13:22:38Z

0xc0170 - can do, although FlashIAP not being a singleton is surely a bug (allocated two instances by mistake in NVStore tests, and the results were erroneous).
In addition, didn't see any usage of FlashIAP anywhere else in the mbed-os code, so impact was only on FlashIAP driver itself (and its test).

davidsaada · 2018-02-20T14:22:29Z

0xc0170 - Not sure if you accepted my answer or not (did you?), but if you did - please don't merge the PR yet. Some of my NVStore tests fail with this change, and I want to be certain that it's not a bug here.

0xc0170 · 2018-02-20T14:51:49Z

0xc0170 - can do, although FlashIAP not being a singleton is surely a bug (allocated two instances by mistake in NVStore tests, and the results were erroneous).

Still this adds new API thus should be treated separately. Let's split it

davidsaada · 2018-02-20T15:19:26Z

0xc0170 - False alarm. Was a problematic board. All's well.

0xc0170 · 2018-02-20T16:49:43Z

drivers/FlashIAP.h

@@ -40,6 +40,13 @@ namespace mbed {
 */
 class FlashIAP : private NonCopyable<FlashIAP> {
 public:
+
+/**


this should be still removed (you can rebase to clean up the history to have only 1 commit here)

0xc0170 · 2018-02-20T16:51:15Z

drivers/FlashIAP.cpp

        return -1;
    }

    int ret = 0;
    _mutex->lock();
-    if (flash_program_page(&_flash, addr, (const uint8_t *)buffer, size)) {
-        ret = -1;
+    while (size) {


while reviewing this, this is similar to #6077 (this is however only program, but erase should do the same) ?

Not sure I understand the comment. Erase seems to be following a similar algorithm as program now. What's the problem?

Oh I see the problem now.
My thought here is that the entire check should just be eliminated. Why not erase everything that's in the range, and let the driver worry about alignments, not the user.

So if I understand correctly, an unaligned erase does this?

--- sector 0 --- -> --- sector 0 --- data data data data data data --- sector 1 --- --- sector 1 --- data ff > data ff unaligned | data ff erase | --- sector 2 --- --- sector 2 --- > data ff data ff data ff

That seems like the sort of thing we'd want to error on.

As said, don't think there should even be an unaligned erase check. Don't even know what it stands for.

Let me rephrase: I believe the only acceptable validation here is to check that the address is on a sector boundary, and that the size is a sum of all the consecutive sector sizes. The only way to achieve this is by iterating on the sectors for the validation, and then iterating again for the erase action. It's either that or check nothing.

Ah ok. Don't let me block this pr then.

Could we check that address + size is on a sector boundary? Would that be cheap?

Yeah, figured this out after writing the answer. This would be the correct check.

geky

Looks good so far, left a few comments though they may just be me misunderstanding.

geky · 2018-02-20T17:46:21Z

drivers/FlashIAP.cpp

+            buf = page_buf;
+            prog_size = page_size;
+        }
+        else {


style nit: else should be on same line as }

geky · 2018-02-20T17:50:16Z

drivers/FlashIAP.cpp

+    const uint8_t *buf = (uint8_t *) buffer;
+
+    // addr should be aligned to page size
+    if (!is_aligned(addr, page_size) || (!buffer)) {


Do we not still need to check that size is aligned and that program does not go past end of the device?

The page alignment check here is for the address only. Don't think we need the size alignment check. Adding such a check would require the user to split the writes to the aligned and unaligned ones, which is basically what the new code does.
Regarding the device end check - good thought. Will add that.

geky · 2018-02-20T17:55:50Z

tools/toolchains/gcc.py

@@ -26,8 +26,7 @@ class GCC(mbedToolchain):
    LIBRARY_EXT = '.a'

    STD_LIB_NAME = "lib%s.a"
-    DIAGNOSTIC_PATTERN = re.compile('((?P<file>[^:]+):(?P<line>\d+):)(\d+:)? (?P<severity>warning|[eE]rror|fatal error): (?P<message>.+)')
-    INDEX_PATTERN  = re.compile('(?P<col>\s*)\^')
+    DIAGNOSTIC_PATTERN = re.compile('((?P<file>[^:]+):(?P<line>\d+):)(?P<col>\d+):? (?P<severity>warning|[eE]rror|fatal error): (?P<message>.+)')


What's the tools change? @theotherjimmy are you able to double check this?

Thanks for the spot! see my comment below.

theotherjimmy · 2018-02-20T18:01:58Z

@davidsaada You have a rebase of a commit on master (1290863ea7973d7862bc006833cf519100a8a39e) in your PR. Please remove this commit from the series.

davidsaada · 2018-02-20T19:17:18Z

Argh. Probably happened after the squash and force push. Thanks @theotherjimmy.

davidsaada · 2018-02-21T08:47:03Z

/morph build

0xc0170 · 2018-02-21T08:59:45Z

/morph build

mbed-ci · 2018-02-21T09:23:00Z

Build : SUCCESS

Build number : 1194
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6140/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-02-21T09:36:15Z

Build : SUCCESS

Build number : 1195
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6140/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

0xc0170 · 2018-02-21T09:26:11Z

drivers/FlashIAP.cpp

@@ -31,6 +33,8 @@ namespace mbed {

 SingletonPtr<PlatformMutex> FlashIAP::_mutex;

+#define BLANK 0xFF


Please use more descriptive name (BLANK can come from targets code and can have diff value), FLASHIAP_BLANK or similar would be better

0xc0170 · 2018-02-21T09:33:35Z

drivers/FlashIAP.h

@@ -128,6 +128,7 @@ class FlashIAP : private NonCopyable<FlashIAP> {
    bool is_aligned_to_sector(uint32_t addr, uint32_t size);

    flash_t _flash;
+    uint8_t *page_buf;


_ prefix for private members

Thanks, forgot that.

0xc0170 · 2018-02-21T09:35:53Z

drivers/FlashIAP.cpp

+        chunk = std::min(current_sector_size - (addr % current_sector_size), size);
+        if (chunk < page_size) {
+            memcpy(page_buf, buf, chunk);
+            memset(page_buf+chunk, BLANK, page_size-chunk);


to be: memset(page_buf + chunk, BLANK, page_size - chunk); (spaces around operators as line 111

0xc0170 · 2018-02-21T09:53:48Z

drivers/FlashIAP.cpp

        return -1;
+    } else if (erase_end_addr < flash_end_addr){
+        uint32_t following_sector_size = flash_get_sector_size(&_flash, erase_end_addr);


is this "following" sector or the "last sector size to be erased"? From the code I read it as it is the last sector to be erased.

@LiyouZhou please review

No. Following means the one after this range. Couldn't find a better name (any recommendation would be welcome).

We are making the assumption that: flash sector always start on an address that is aligned to the flash sector size. This is true for all platfroms that I know but I don't know if it is generally true.

I think this will work for the issue i raised. Can we have a testcase to make sure there is no regression?

I also never saw cases where sector addresses weren't aligned to their own sizes. This is pretty easy to fix (subtract flash start from relevant cases). Not sure it justifies a special test (if I understood you correctly).

I meant a test for trying to erase 2 sectors of different sizes at once. basically the situation i describe here: #6077

Oh I see. The NUCLEO_F410RB has two last sectors having different sizes (16KB and 64KB). I can change my new test to erase the last two sectors in one call, and this would cover the case.

Fixed the test to erase the two last sectors in one call. This will now check your described scenarios in boards whose last two sector sizes are different (there are a few).

@LiyouZhou Can you review the latest changes and leave feedback here?

@LiyouZhou Would appreciate a quick review, as it seems your review is all that's needed to apply this PR.

0xc0170 · 2018-02-21T10:00:17Z

TESTS/mbed_drivers/flashiap/main.cpp

+    }
+
+    address += sector_size - 3 * page_size;
+    uint32_t aligned_prog_size = 7 * page_size;


some devices might fail to allocate 7page_size (if we consider that data are deleted on line 124, this means we are allocating double than that, results in 14page_size?

Lets review the test results for this test

Maybe I'm mistaken, but AFAIK page sizes are typically 8 bytes. Is this problematic?

I recall there was a problem but was probably with sectors rather than pages (as page size that is returned is the smallest unit thta can be written, thus in most cases would be 8 bytes).

Just checking that this will be ok for all targets

Appears you were right. Boards (namely ARCH_PRO) have a 1KB page size. Modifying the test accordingly.

0xc0170 · 2018-02-21T10:51:44Z

Support programming across sectors.
Support program size not aligned to page size.

One last check, do documentation need any updating with this fixes? It would be good if this is captured in the methods documentation what is supported and what is not. I read t he report this week related to program/erase missing this.

davidsaada · 2018-02-21T11:45:47Z

Looking at the handbook, the md file doesn't require any changes. The code (and derived Doxygen documentation) , however, needs a bit of modifications in the program API: It says that the size needs to be aligned to sector and page sizes. The latter isn't true any more, but the former was never true. Will change that in the code, but Doxy needs to be regenerated.

mbed-ci · 2018-02-21T12:40:20Z

Exporter Build : SUCCESS

Build number : 868
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/6140/

mbed-ci · 2018-02-21T12:40:23Z

Exporter Build : SUCCESS

Build number : 868
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/6140/

mbed-ci · 2018-02-21T13:05:00Z

Test : FAILURE

Build number : 997
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6140/997

mbed-ci · 2018-02-21T14:24:03Z

Test : FAILURE

Build number : 998
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6140/998

c1728p9 · 2018-02-22T19:29:21Z

drivers/FlashIAP.cpp

@@ -31,6 +33,8 @@ namespace mbed {

 SingletonPtr<PlatformMutex> FlashIAP::_mutex;

+#define FLASHIAP_BLANK 0xFF


Many of the STM32L devices have an erased value of 0x00 rather than 0xFF. (the only reason I know this is because I came across it today while adding DAPLink support to the STM32L151CBU6A) The HAL API should probably to be modified to expose this information if it is used as this level.

In this case it looks like this value may be used just as a pad value. If so you might want to rename it so it isn't confused with the erased/blank value.

This is indeed used just once as a PAD value in the driver, so the define will be removed.
The driver doesn't assume that blank values are 0xFF, but other code (such as NVStore) does, so will ask for such an API in HAL. If and when implemented, it should be exposed via this driver.

c1728p9 · 2018-02-22T19:54:39Z

drivers/FlashIAP.cpp

+        if (chunk < page_size) {
+            memcpy(_page_buf, buf, chunk);
+            memset(_page_buf + chunk, FLASHIAP_BLANK, page_size - chunk);
+            buf = _page_buf;


buf is clobbered here and never restored. It isn't actually a problem though since it should only happen on the last iteration of the loop. It may not hurt to add another variable, something like prog_buf to go along with prog_size so buf doesn't need to be clobbered.

Good idea. This was intentional knowing it's the last iteration, but it can be missed if the code is modified later on.

cmonr · 2018-02-22T20:29:42Z

@davidsaada Holding off on the merge until the commit that addresses @c1728p9's review comes in.

Separate note, was this PR ever split up, according to @0xc0170's request? (#6140 (comment))

davidsaada · 2018-02-22T20:33:01Z

@cmonr No problem. Committing a quick fix soon. And yes - the PR was split up. Will issue the second part once this one is merged.

- Support programming across sectors. - Support program size not aligned to page size. - Fix validations on sector erase.

0xc0170 · 2018-02-22T21:29:57Z

/morph build

mbed-ci · 2018-02-22T21:49:04Z

Build : SUCCESS

Build number : 1224
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6140/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-02-23T03:55:43Z

Test : SUCCESS

Build number : 1025
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6140/1025

mbed-ci · 2018-02-23T03:55:57Z

Exporter Build : FAILURE

Build number : 893
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/6140/

davidsaada · 2018-02-23T05:45:21Z

/morph build

davidsaada · 2018-02-23T05:47:03Z

Started morph build again. That exporter build failure seems like a setup glitch.

mbed-ci · 2018-02-23T06:16:38Z

Build : SUCCESS

Build number : 1226
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6140/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-02-23T07:26:45Z

Test : SUCCESS

Build number : 1027
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6140/1027

mbed-ci · 2018-02-23T08:08:06Z

Exporter Build : SUCCESS

Build number : 896
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/6140/

0xc0170

Testing the master protection against merging with request changes status (will revert asap)

0xc0170 added the needs: review label Feb 20, 2018

0xc0170 requested a review from c1728p9 February 20, 2018 13:11

0xc0170 reviewed Feb 20, 2018

View reviewed changes

davidsaada force-pushed the david_flashiap_xsectors branch from 59275d8 to 20c6799 Compare February 20, 2018 17:17

geky reviewed Feb 20, 2018

View reviewed changes

davidsaada force-pushed the david_flashiap_xsectors branch 3 times, most recently from a0d5fd7 to 20646dc Compare February 20, 2018 23:00

davidsaada mentioned this pull request Feb 21, 2018

Add NVStore (A.K.A SOTP) feature #5900

Merged

0xc0170 reviewed Feb 21, 2018

View reviewed changes

0xc0170 added needs: work and removed needs: review labels Feb 21, 2018

c1728p9 approved these changes Feb 22, 2018

View reviewed changes

cmonr added ready for merge needs: CI and removed needs: review ready for merge labels Feb 22, 2018

FlashIAP driver modifications:

0aeeece

- Support programming across sectors. - Support program size not aligned to page size. - Fix validations on sector erase.

davidsaada force-pushed the david_flashiap_xsectors branch from 326ac03 to 0aeeece Compare February 22, 2018 20:50

0xc0170 added ready for merge release-version: 5.7.6 and removed needs: CI labels Feb 23, 2018

0xc0170 requested changes Feb 23, 2018

View reviewed changes

0xc0170 approved these changes Feb 23, 2018

View reviewed changes

adbridge added release-version: 5.8.0-rc1 and removed release-version: 5.7.6 labels Feb 23, 2018

0xc0170 merged commit 1079767 into ARMmbed:master Feb 23, 2018

0xc0170 removed the ready for merge label Feb 23, 2018

davidsaada deleted the david_flashiap_xsectors branch March 20, 2018 10:34

davidsaada mentioned this pull request Jun 18, 2018

FlashIAP does not deal with erasing sectors correctly #6077

Closed

yogeshk19 mentioned this pull request Jun 25, 2019

NRF52840: Custom bootloader hangs when calling mbed_start_application on a re-flash of new firmware. #10890

Closed

		@@ -31,6 +33,8 @@ namespace mbed {

		SingletonPtr<PlatformMutex> FlashIAP::_mutex;

		#define BLANK 0xFF

		@@ -31,6 +33,8 @@ namespace mbed {

		SingletonPtr<PlatformMutex> FlashIAP::_mutex;

		#define FLASHIAP_BLANK 0xFF

FlashIAP driver modifications #6140

FlashIAP driver modifications #6140

Conversation

davidsaada commented Feb 20, 2018 • edited Loading

Description

Status

0xc0170 commented Feb 20, 2018 • edited Loading

davidsaada commented Feb 20, 2018

davidsaada commented Feb 20, 2018

0xc0170 commented Feb 20, 2018

davidsaada commented Feb 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

geky left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theotherjimmy commented Feb 20, 2018

davidsaada commented Feb 20, 2018

davidsaada commented Feb 21, 2018

0xc0170 commented Feb 21, 2018

mbed-ci commented Feb 21, 2018

Triggering tests

mbed-ci commented Feb 21, 2018

Triggering tests

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xc0170 commented Feb 21, 2018

davidsaada commented Feb 21, 2018

mbed-ci commented Feb 21, 2018

mbed-ci commented Feb 21, 2018

mbed-ci commented Feb 21, 2018

mbed-ci commented Feb 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmonr commented Feb 22, 2018 • edited Loading

davidsaada commented Feb 22, 2018 • edited Loading

0xc0170 commented Feb 22, 2018

mbed-ci commented Feb 22, 2018

Triggering tests

mbed-ci commented Feb 23, 2018

mbed-ci commented Feb 23, 2018

davidsaada commented Feb 23, 2018

davidsaada commented Feb 23, 2018

mbed-ci commented Feb 23, 2018

Triggering tests

mbed-ci commented Feb 23, 2018

mbed-ci commented Feb 23, 2018

0xc0170 left a comment

Choose a reason for hiding this comment

davidsaada commented Feb 20, 2018 •

edited

Loading

0xc0170 commented Feb 20, 2018 •

edited

Loading

cmonr commented Feb 22, 2018 •

edited

Loading

davidsaada commented Feb 22, 2018 •

edited

Loading