Add special handling for "/default" filesystem #7924

kjbracey · 2018-08-29T12:49:49Z

Description

Allow a FileBase (normally a FileSystemLike) to be set as the default, so it can be looked up as "/default" as well as its actual name.

Intended to be used to extend the implementation in #7774. Once that is merged, we can add a commit to do that in this PR.

Expected pattern is

FileSystem *FileSystem::get_default_instance()
{
    static FATFileSystem sdcard("sd", sdcard_block_device);
    sdcard.set_as_default();
    return &sdcard;
}

This will allow simple code using "/default" to interwork with custom-configured setups preferring to use explicit mount names.

Pull request type

[ ] Fix
[ ] Refactor
[ ] Target update
[X] Functionality change
[ ] Breaking change

yossi2le · 2018-09-02T12:27:39Z

The code is OK but, I do have a problem with doing things under the hood. This feature is actually aiming to cloak the system paths. It actually means that while an application assumes it writes to default/ path, it will go to different one and not exactly a predictable one.

offirko · 2018-09-02T13:00:52Z

platform/FileBase.cpp

+
+    if (_default == this) {
+        _default == NULL;
+    }

    if (getPathType() == FilePathType) {


This part was not locked by _mutex in the original code. Is it locked now by ScopedLock ?
If so, then note that 'remove_filehandle()' uses a lock of its own: filehandle_mutex->lock();
Consider the risk of deadlock.

Ta for pointing that out - hadn't considered it. It seems okay, as mbed_retarget.cpp does not call out to File objects while holding its filehandle_mutex - it's lower-level.

0xc0170 · 2018-09-02T18:30:11Z

This needs review and consideration, as its functionality change, might be considered for 5.11 rather?

kjbracey · 2018-09-03T07:24:10Z

The cloaking is already happening manually in #7774 - a filing system is being designated as the default, and is responding to "/default", so someone requesting via that name already doesn't know what it will be.

This improves the situation by allowing the device to still have a proper name. Without this, and just #7774, you might have this situation:

Platform A:

/default = FAT sd card
/littlefs = onboard storage

Platform B:

/default = onboard storage
/sd = FAT sd card

Like that, a portable program wanting to explicitly write to SD is going to struggle, as there's no consistent name - the default concept has got in the way.

With this change, we can do this, which seems better:

Platform A:

/default or /sd = FAT sd card
/littlefs = onboard storage

Platform B:

/sd = FAT sd card
/default or /littlefs = onboard storage

Programs can now reasonably request "/sd", "/littlefs" or "/default".

kjbracey · 2018-09-03T07:26:10Z

This needs review and consideration, as its functionality change, might be considered for 5.11 rather?

Would be nice to get it in, as I fear that without it I fear #7774 may somewhat distort the effective API. A default shouldn't get in the way of explicit selection.

But on the other hand, we'd still need the extension to 7774 to do the renaming as per the previous comment, so maybe it's too late now.

0xc0170 · 2018-09-03T07:30:06Z

cc @davidsaada @dannybenor

davidsaada

Looks good to me.

yossi2le · 2018-09-03T09:23:52Z

@kjbracey-arm . Ok, I see your point and agreed.
I just think that in that case, you should add the call to set_as_default also to the default implementation found in SystemSorage.cpp.

0xc0170 · 2018-09-03T09:49:23Z

/morph build

mbed-ci · 2018-09-03T12:56:25Z

Build : SUCCESS

Build number : 2995
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/7924/

Triggering tests

/morph test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-09-03T17:36:16Z

Test : FAILURE

Build number : 2758
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/7924/2758

mbed-ci · 2018-09-03T18:21:17Z

Exporter Build : SUCCESS

Build number : 2606
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/7924/

0xc0170 · 2018-09-04T06:20:12Z

/morph test

kjbracey · 2018-09-04T08:39:02Z

I just think that in that case, you should add the call to set_as_default also to the default implementation found in SystemSorage.cpp.

Just been creating a PR to do that, but I've noticed that 7774 went in using "/fs", rather than "/default", My comment calling "/fs" a tautology got thumbs up so I assumed you went with that.

Not sure what to do now. Here are the 3 possibilities I see, in my order of preference:

Keep this as is, change tests (and docs?) to reference "/default".
Extend this to have both /default and /fs as aliases.
Change this to set /fs as the alias instead of /default.

yossi2le · 2018-09-04T09:03:06Z

I just think that in that case, you should add the call to set_as_default also to the default >>implementation found in SystemSorage.cpp.

Just been creating a PR to do that, but I've noticed that 7774 went in using "/fs", rather than "/default", >My comment calling "/fs" a tautology got thumbs up so I assumed you went with that.

Sorry I missed the fs/ I have thought the big issue was to move back to the original naming pattern and use get_default_instance()

Not sure what to do now. Here are the 3 possibilities I see, in my order of preference:

Keep this as is, change tests (and docs?) to reference "/default".
Extend this to have both /default and /fs as aliases.
Change this to set /fs as the alias instead of /default.

It very easy to change the default to default/. need to fix SystemStorage.cpp which I believe should be fixed anyway to include a call to set_as_default().
and also change the tests at features/storage/TESTS/filesystem/general_filesyste/main.cpp to use default/ and not fs/

However, your solution does work even if it left "fs/" and "default/" will be an alias of "fs/". i thought this was the all idea?

kjbracey · 2018-09-04T09:11:08Z

Just created #7977, which extends on this and flips everything to "/default". If you're okay with that, it would be awesome to try to get that into this release. I think the final result should make more sense to the end users.

kjbracey · 2018-09-04T09:26:50Z

However, your solution does work even if it left "fs/" and "default/" will be an alias of "fs/". i thought this was the all idea?

The idea was that we would use the mechanism maintain "physical" names like "/sd" or "/flash", in addition to supporting logical names. The initial only logical name is "/default", but we could have others in future with a general scheme. I want the devices' primary name - visible in config and errors - to be their physical one, so instantiate as "sd".

We could maintain "/fs" as a duplicate logical name for backwards compatibility, but at this point it would be cleaner to quickly switch before release, if people are okay with "/default".

If you look at 7977's final version, I think we have a great level of extra clarity on what's happening:

static LittleFileSystem flash("flash", BlockDevice::get_default_instance());
flash.set_as_default();

return &flash;

or

static FATFileSystem sdcard("sd", BlockDevice::get_default_instance());
sdcard.set_as_default();

return &sdcard;

The default mechanism is, intentionally, an obfuscation, so we've got to take care we don't "over-obfuscate". We still need some real names backing the system - even if "good code" doesn't reference it directly, having standard conventions allowing people to re-use config files specifying "/sd/xxx" is worthwhile.

mbed-ci · 2018-09-04T09:50:40Z

Test : SUCCESS

Build number : 2766
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/7924/2766

yossi2le · 2018-09-04T10:35:33Z

@kjbracey-arm it sounds Ok to me.

0xc0170 · 2018-09-04T14:29:53Z

@teetak01, @JaniSuonpera, @JanneKiiskila - need your input on this and its part 2 #7977. Does what I'm proposing make sense to you as a user, and would you be okay with this going into rc2?

As soon as we get positive feedback, this will be ready for integration

geky · 2018-09-04T18:26:12Z

This is one of the APIs that needs a long-term change. My current thoughts are to split the FileBase out of the inheritance tree into it's own MountPoint class (the current FileBase class has already caused the duplication of all major file system interfaces, FileLike/FileHandle, FileSystemLike/FileSystemHandle).

For the short term why not just add "/default" to the config system? "platform.default_mount_point" or something?

I'm also not sure it's a big rush because moving "/default" to the config system should be a backwards-compatible change. But it's possible I'm missing something.

geky · 2018-09-04T18:33:24Z

platform/FileBase.cpp

    return NULL;
 }

 FileBase *FileBase::get(int n)
 {
-    _mutex->lock();
+    ScopedLock<PlatformMutex> lock(*_mutex);


This is the wrong PR to change these locking calls.

I'm not opposed to this change, but every change invites bugs.

You're not wrong - I certainly wouldn't do this in a patch/fix. Probably shouldn't be doing it in rc2 either.

I can undo this, but as the current test set has passed, and we might be accepting this version otherwise, is it worth restarting CI just to take it out? @0xc0170 ?

I'll prepare a removal anyway in case I do another spin, but won't push yet.

I agree not to block it now, it's just not a good habit to get into

Actually another CI round is required anyway for part 2 - #7977, so we could still change it.

geky · 2018-09-04T18:35:50Z

platform/SingletonPtr.h

+     * @returns
+     *   A reference to the singleton
+     */
+    T &operator*()


Isn't this taken care of by the T *operator->() operator?

No - the alternative here would be lock(*_mutex.get()), which seemed unnecessarily awkward.

Ah you're right, I though operator-> was full magic, but it's only half magic 👍

That does cover the "I want a reference" case, but I wonder if we should also have a conversion operator to T * for people who want the pointer. Does the get() really have to be explicit? Is there any C++ trap there for the conversion operator?

JaniSuonpera

Tested changes with mbed-os-5.10.

kjbracey · 2018-09-05T09:04:50Z

I agree that FileBase would be due for a significant rework - not clear how much a File and FilingSystem have in common - and a proper mount point system could lead to that.

For the short term why not just add "/default" to the config system? "platform.default_mount_point" or something?

So change the model "get_default_instance()" implementations to name them as "/sd" or "/flash", and make the JSON have a setting telling you what that name is?

That would be possible, but it's not quite as flexible - "FileSystem::get_default_instance()" is a run-time call, so the default mount point can be a run-time decision (eg depending on if an SD card is present), or it can be based on conceptual #ifdefs like the default weak version in #7774. A config name specification couldn't mirror that. And I was trying to preserve the design choice of #7774 that the default mount point has a well-known name - I think it's a useful convenience.

You could have a separate FileSystem::get_default_mount_name() call alongside the get_default_instance(), but it's less convenient - you have to start doing the strcats.

I'm imagining app config files like

"key-storage" = "/default/keys/",
"image-storage" = "/default/images/",
"target_overrides" = {
       "platform_x" = {
            "key-storage" = "/secure/keys/",
            "image-storage" = "/flash/images/"

That app can run on anything that's got some sort of general-purpose filing system, but for specific platforms it has specific set-up code and configuration.

My view is that these two modes of operation - explicitly specified and "default" for the mount point should be able to co-exist, and 7774 as it stands makes that tricky - if you make the sdcard be the default you can't any longer explicitly refer to it as /sd, and if you name it as "/sd" for explicit configuration, you can't coexist with code expecting to just use "/default" - you have to explicitly configure everything or nothing.

The somewhat app-specific boot-up code for platform_x to back that would contain

SecureFS secure("secure", ...);
LittleFS flash("flash", ....);
flash.set_as_default();

FilingSystem *FilingSystem::get_default_instance() {
    return &flash;
}

So all filing systems are present and named for config to refer to, and one of them is still marked as default so the basic get_default_instance API and "/default" remains functional. And this setup has to be somewhat app-specific as you have to know which filing systems are worth instantiating for a particular app, and you may have app-specific FS configuration requirements (eg speed vs lifetime tuning parameters).

Allow a FileBase (normally a FileSystemLike) to be set as the default, so it can be looked up as "/default" as well as its actual name.

geky · 2018-09-05T10:31:29Z

That would be possible, but it's not quite as flexible - "FileSystem::get_default_instance()" is a run-time call, so the default mount point can be a run-time decision

Though note that choosing the name of the mount point is not a runtime call:

FileSystem *fs = FileSystem::get_default_instance();

There's nothing there that specifies what the FS's name is. At the moment this is a compile time decision as far as I can tell.

My view is that these two modes of operation - explicitly specified and "default" for the mount point should be able to co-exist, and 7774 as it stands makes that tricky

I agree, but I don't think we're there yet. At the moment the underlying retarget code only supports 1 mount point per FS.

Another short-term option, what if we add "/default" to the config system, and make the default "/default" NULL. This effectively does not mount the FS to the tree.

That would leave things open to add multiple mount points and reintroduce "/default" properly next release.

kjbracey · 2018-09-05T11:09:00Z

There's nothing there that specifies what the FS's name is. At the moment this is a compile time decision as far as I can tell.

This came up during review, and the agreement was that as there was no way to pass a FilingSystem to fopen etc, to make this usage possible there would instead be a convention, which those initial implementations are setting down - the default filing system has a well-known name, so after making that call, code or config can assume the name.

 // make sure we have it, and it's initialised and linked in
 FilingSystem *fs = FilingSystem::get_default_instance();
 if (!fs) {
       error("no filing system!");
 }
 // use it - either via well-known name (but wouldn't work without the call above)
 FILE *log = fopen("/default/log");
 // or explicitly from return value
 File *log = fs->open("log");

So given that, that means that the sdcard is called "default" (or "fs"), not "sd", because without this PR there's no way to present "default" as an alias. I'm happy with the "default" name, but I'm regretting the loss of "sd".

The other possibility here would be to junk the idea of a well-known name and make code extract it. I'm not sure this option was properly considered:

 FilingSystem *fs = FilingSystem::get_default_instance();   
 const char *fs_name = fs->getName();

 sprintf(fname, "/%s/log", fs_name)
 FILE *log = fopen(fname);

I'd be okay with that in code, but does preclude the short-cut of just doing "/default/foo" in the config file - using the default and using something explicit would actually be different code at each call site, not just flipping config between "/default/foo" and "/sd/foo". I do like the concept of it ultimately always just coming down to a string constant.

Another short-term option, what if we add "default" to the config system, and make the default "default" NULL. This effectively does not mount the FS to the tree.

Sorry, you've lost me now. Can you be more specific?

That would leave things open to add multiple mount points properly next release.

Once something like that is in place, the set_as_default() call just becomes a wrapper around mount(fs, "/default"), and /default would be just 1 well-known mount name.

"/fs" is a tautology - not a good name for the default filing system, as whereever we use it, we know we're specifying a filing system. Rename to "/default".

Make the built-in FileSystem::get_default_instance() implementation instantiate storage as "flash" or "sd", with "default" as an alias. This will aid interworking between simple and advanced filesystem code on various platforms. The expectation is that the names "sd" or "flash" will be always available if the device is available and configured, regardless of what "default" represents.

geky · 2018-09-05T11:58:37Z

the default filing system has a well-known name, so after making that call, code or config can assume the name.

Doesn't that still make this a compile time decision?

I'm happy with the "default" name, but I'm regretting the loss of "sd".

This is a new API, and it's provided at the target level. Unless this target is mounting the FS as both "default" and "sd" (which I don't believe it is?), we won't have "sd".

Unless the user declares an FS as "sd", but then it won't be a part of the get_default_instance.

Actually that raises a separate question. Does FileBase::set_as_default change what FileSystem::get_default_instance returns?

"default" NULL.... Sorry, you've lost me now. Can you be more specific?

FileBase supports NULL, which FileSystem::get_default_instance() does not need to create a mount point. Though it sounds like from your comment this isn't what we're looking for.

Once something like that is in place, the set_as_default() call just becomes a wrapper around mount(fs, "/default"), and /default would be just 1 well-known mount name.

True, I originally thought the cost was 1 mount point per filesystem, but after looking at it again we only need a global one. So it's a minor cost.

kjbracey · 2018-09-05T12:23:57Z

Doesn't that still make this a compile time decision?

I would say it's a design-time decision, if it's a decision at all - it's was intended to be an API invariant, not something that would change between programs, builds or platforms.

Unless the user declares an FS as "sd", but then it won't be a part of the get_default_instance.

But why not? Why can't someone have 2 or 3 explicitly-mounted filing systems, and have one of them be the default? That's what I've shown in several of my examples, and that is the motivation for my wanting a change here. See this comment here: #7924 (comment)

I don't want there to be this exclusion - either you have exactly 1 filing system which is accessible via the "get_default" or you have multiple, but there's no default. The very name "get_default" suggests it's 1 of potentially several. For the simple case there is this default weak instantiation of "get_default_instance" that spawns the (only?) FS on demand, but the intent is that you can override to point at something you've explicitly set-up yourself, as per this comment: #7924 (comment)

But when you do that, the current API contract is that the thing you point the get_default_instance to has to be given the well-known name. Without aliases you have the situation I don't want which is "the sdcard is conventionally mounted as /sd, unless it's the default." Surely it's simpler that it's conventionally explicitly accessible as "/sd", whether or not it's the default?

Actually that raises a separate question. Does FileBase::set_as_default change what FileSystem::get_default_instance returns?

No, the causality is in the other direction - whoever is providing FileSystem::get_default_instance() can call FileBase::set_as_default() on what they provide to satisfy the "must be named as /default" postcondition. Maybe the method could have a clearer name - I did consider "name_as_default()".

The simple get_default_instance() implementation has to be a "pull" API that instantiates on-demand for linker exclusion. If you had it just return what was pushed in by "set_as_default", then the linker would include a filing system due to the set whether or not anyone called get_default_instance.

In the more complex examples I've given previously, the assumption is that you have some application knowledge so you know what filing systems are worth mounting and hence pulling into the image up-front.

kjbracey · 2018-09-05T13:21:38Z

I needed to correct an error in the string comparison, so redone. This time without the ScopedLock and SingletonPtr changes, and adding in the extra completion from #7977.

cmonr · 2018-09-05T15:40:45Z

@geky Alright with the spinout of #8001?

cmonr · 2018-09-05T15:49:13Z

/morph build

geky

Yeah, thanks for taking those out 👍

My comments are concerns but not blockers

mbed-ci · 2018-09-05T16:38:20Z

Build : SUCCESS

Build number : 3014
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/7924/

Triggering tests

/morph test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-09-05T19:01:42Z

Exporter Build : SUCCESS

Build number : 2625
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/7924/

mbed-ci · 2018-09-05T20:22:10Z

Test : SUCCESS

Build number : 2780
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/7924/2780

0xc0170 added the needs: review label Aug 29, 2018

0xc0170 requested a review from a team August 29, 2018 12:54

kjbracey force-pushed the filebase_default branch from c048d92 to 6a2d56a Compare August 29, 2018 13:41

cmonr added the release-version: 5.10.0-rc1 label Aug 29, 2018

kjbracey force-pushed the filebase_default branch from 6a2d56a to da4ae15 Compare August 31, 2018 12:14

offirko reviewed Sep 2, 2018

View reviewed changes

davidsaada approved these changes Sep 3, 2018

View reviewed changes

offirko approved these changes Sep 3, 2018

View reviewed changes

adbridge added needs: CI and removed needs: review labels Sep 3, 2018

kjbracey mentioned this pull request Sep 4, 2018

Take default FS alias into use #7977

Closed

0xc0170 added release-version: 5.10.0-rc2 and removed release-version: 5.10.0-rc1 labels Sep 4, 2018

geky reviewed Sep 4, 2018

View reviewed changes

JaniSuonpera approved these changes Sep 5, 2018

View reviewed changes

Add special handling for "/default" filesystem

a67f098

Allow a FileBase (normally a FileSystemLike) to be set as the default, so it can be looked up as "/default" as well as its actual name.

kjbracey added 2 commits September 5, 2018 14:18

Change default filesystem name to /default

fd4b387

"/fs" is a tautology - not a good name for the default filing system, as whereever we use it, we know we're specifying a filing system. Rename to "/default".

kjbracey force-pushed the filebase_default branch from da4ae15 to c0dbc49 Compare September 5, 2018 13:19

kjbracey mentioned this pull request Sep 5, 2018

Add * operator to SingletonPtr #8001

Merged

cmonr added needs: CI and removed needs: review labels Sep 5, 2018

geky approved these changes Sep 5, 2018

View reviewed changes

cmonr added ready for merge and removed needs: CI labels Sep 6, 2018

cmonr merged commit 6c30b2f into ARMmbed:master Sep 6, 2018

0xc0170 removed the ready for merge label Sep 6, 2018

kjbracey deleted the filebase_default branch September 7, 2018 09:13

korjaa mentioned this pull request Oct 17, 2018

Added missing _mutex->unlock() to FileBase::lookup(). #8451

Merged

Add special handling for "/default" filesystem #7924

Add special handling for "/default" filesystem #7924

Conversation

kjbracey commented Aug 29, 2018 • edited Loading

Description

Pull request type

yossi2le commented Sep 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xc0170 commented Sep 2, 2018

kjbracey commented Sep 3, 2018

kjbracey commented Sep 3, 2018 • edited Loading

0xc0170 commented Sep 3, 2018

davidsaada left a comment

Choose a reason for hiding this comment

yossi2le commented Sep 3, 2018

0xc0170 commented Sep 3, 2018

mbed-ci commented Sep 3, 2018

Triggering tests

mbed-ci commented Sep 3, 2018

mbed-ci commented Sep 3, 2018

0xc0170 commented Sep 4, 2018

kjbracey commented Sep 4, 2018

yossi2le commented Sep 4, 2018

kjbracey commented Sep 4, 2018

kjbracey commented Sep 4, 2018 • edited Loading

mbed-ci commented Sep 4, 2018

yossi2le commented Sep 4, 2018

0xc0170 commented Sep 4, 2018

geky commented Sep 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

geky Sep 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JaniSuonpera left a comment

Choose a reason for hiding this comment

kjbracey commented Sep 5, 2018

geky commented Sep 5, 2018 • edited Loading

kjbracey commented Sep 5, 2018

geky commented Sep 5, 2018 • edited Loading

kjbracey commented Sep 5, 2018

kjbracey commented Sep 5, 2018

cmonr commented Sep 5, 2018

cmonr commented Sep 5, 2018

geky left a comment

Choose a reason for hiding this comment

mbed-ci commented Sep 5, 2018

Triggering tests

mbed-ci commented Sep 5, 2018

mbed-ci commented Sep 5, 2018

kjbracey commented Aug 29, 2018 •

edited

Loading

kjbracey commented Sep 3, 2018 •

edited

Loading

kjbracey commented Sep 4, 2018 •

edited

Loading

geky Sep 5, 2018 •

edited

Loading

geky commented Sep 5, 2018 •

edited

Loading

geky commented Sep 5, 2018 •

edited

Loading