Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Little FS improvements and IFS revisions to support user metadata #2308

Merged
merged 11 commits into from
Apr 20, 2021

Conversation

mikee47
Copy link
Contributor

@mikee47 mikee47 commented Apr 19, 2021

Main changes

This PR adds the following methods to IFileSystem:

  • setProfiler

This allows applications to hook into the low-level read/write/erase calls for debugging and profiling.

  • fsetxattr
  • fgetxattr
  • setxattr
  • getxattr

The setacl, settime, setattr and setcompression methods have been moved to FileSystem and
call the above new methods to do their work.

Additional overloads have been provided in FileSystem (and via methods of File objects):

  • setAttribute
  • getAttribute

These work with file handles or paths and have various convenient overloads.

  • setUserAttribute, getUserAttribute

These are to access user-defined attributes.

For LFS volumes, each item is limited to a maximum size of 1023 bytes.

SPIFFS is less flexible (by design). Metadata is stored in the object headers
(after filename, etc.) and is restricted by the setting of SPIFFS_OBJ_META_LEN.
The first 16 bytes are used for system attributes (e.g. modified time), with user attributes
occupying any remaining space.

The current default setting (16) is unchanged, so if user metadata for SPIFFS volumes
is required then this must be increased accordingly.
Each user attribute requires a 2-byte header (tag + size).

  • fopen
  • fopendir

Removed - see below.

Fixes

  • Revise HYFS to track file path

SPIFFS getFilePath() call sometimes returns weird results.

  • Fix SPIFFS FileSystem::getFilePath return code

  • Fix FileDir weirdness

The generic DirHandle type points to an IFS::FileDir struct, which each filesystem defines internally as required.
It looks like the compiler sometimes mixes these up and causes problems,
though this has only been seen when compiling in debug mode for Host.

So instead, redefine DirHandle as an abstract type, move the FileDir definition inside the appropriate filesystem namespace
and add the GET_FILEDIR() macro to deal with checking and casting.

  • Fix memory leak in test module (thanks valgrind)

  • Set lasterror in File::readContent()

  • Fix FileSystem::makedirs()

  • Fix LFS reported volumesize

Improvements

  • Increase LFS_CACHE_SIZE from 16 to 32 bytes

Results in signficant reduction in read counts

  • Switch to littlefs fork, manage file attributes manually

Only changed attributes (and those with non-default values) get written out
File time only gets updated if file is changed, opening in write mode is insufficient

  • LFS mkdir sets mtime on success

  • LFS mkdir succeeds if directory already exists

  • Add LFS inspect sample

  • Implement Host flush, rename and remove methods

  • Helpers return FileSystem* instead of IFileSystem* for easier use

  • Add optional IFileSystem::setProfiler method to enable flexible debugging and performance evaluation

  • Revise hybrid filesystem to allow use of alternative writeable filesystem (e.g. Little FS)

  • Remove IFileSystem.fopendir(Stat&) and IFileSystem.fopen(Stat&)

The goal of these methods was to provide more efficient means to open files/directories discovered during enumeration.
However, they are difficult to implement effectively and the additional complexity is hard to justify.

POSIX defines a whole raft of such methods such as openat, mkdirat, etc. which add an additional dirfd parameter
to specific the working directory. See https://man7.org/linux/man-pages/man3/open.3p.html for some background.

This may be re-visited in the future if the need arises.

  • Add generic get/set attribute methods

These support LFS file/directory attributes as well as standard attribute types via AttributeTag.

There is no POSIX specification for this but the linux convention (setxattr, etc.) are appropriate.

Change `file id` to `file handle`
Change `file name` to `file path`
Adds complexity for little gain.
…ind)

Parts of oix_hdr not initialised so may leak sensitive information to disk
Fixes

- Revise HYFS to track file path

SPIFFS getFilePath() call sometimes returns weird results.

- Fix SPIFFS  FileSystem::getFilePath return code

- Fix FileDir weirdness

The generic `DirHandle` type points to an `IFS::FileDir` struct, which each filesystem defines internally as required.
It looks like the compiler sometimes mixes these up and causes problems,
though this has only been seen when compiling in debug mode for Host.

So instead, redefine `DirHandle` as an abstract type, move the `FileDir` definition inside the appropriate filesystem namespace
and add the `GET_FILEDIR()` macro to deal with checking and casting.

- Fix memory leak in test module (thanks valgrind)

- Set lasterror in `File::readContent()`

- Fix `FileSystem::makedirs()`

Improvements

- Implement Host `flush`, `rename` and `remove` methods

- Helpers return FileSystem* instead of IFileSystem* for easier use

- Add optional `IFileSystem::setProfiler` method to enable flexible debugging and performance evaluation

- Revise hybrid filesystem to allow use of alternative writeable filesystem (e.g. Little FS)

Changes to core interface

- Remove `IFileSystem.fopendir(Stat&)` and `IFileSystem.fopen(Stat&)`

The goal of these methods was to provide more efficient means to open files/directories discovered during enumeration.
However, they are difficult to implemenent effectively and the additional complexity is hard to justify.

POSIX defines a whole raft of such methods such as `openat`, `mkdirat`, etc. which add an additional `dirfd` parameter
to specific the working directory. See https://man7.org/linux/man-pages/man3/open.3p.html for some background.

This may be re-visited in the future if the need arises.

- Add generic get/set attribute methods

These support LFS file/directory attributes as well as standard attribute types via `AttributeTag`.

There is no POSIX specification for this but the linux convention (setxattr, etc.) are appropriate.
@mikee47 mikee47 changed the title Update/littlefs improvements Little FS improvements and IFS revisions to support user metadata Apr 19, 2021
@slaff slaff added this to the 4.3.1 milestone Apr 19, 2021
@slaff slaff merged commit eaafc21 into SmingHub:develop Apr 20, 2021
@slaff slaff mentioned this pull request Apr 20, 2021
5 tasks
@mikee47 mikee47 deleted the update/littlefs-improvements branch April 20, 2021 08:05
slaff pushed a commit that referenced this pull request Sep 27, 2021
)

**Main changes**

This PR adds the following methods to `IFileSystem`:

* setProfiler

This allows applications to hook into the low-level read/write/erase calls for debugging and profiling.

* fsetxattr
* fgetxattr
* setxattr
* getxattr

The `setacl`, `settime`, `setattr` and `setcompression` methods have been moved to `FileSystem` and
call the above new methods to do their work.

Additional overloads have been provided in FileSystem (and via methods of File objects):

* setAttribute
* getAttribute

These work with file handles or paths and have various convenient overloads.

* setUserAttribute, getUserAttribute

These are to access user-defined attributes.

For LFS volumes, each item is limited to a maximum size of 1023 bytes.

SPIFFS is less flexible (by design). Metadata is stored in the object headers
(after filename, etc.) and is restricted by the setting of `SPIFFS_OBJ_META_LEN`.
The first 16 bytes are used for system attributes (e.g. modified time), with user attributes
occupying any remaining space.

The current default setting (16) is unchanged, so if user metadata for SPIFFS volumes
is required then this must be increased accordingly.
Each user attribute requires a 2-byte header (tag + size).


* fopen
* fopendir

Removed - see below.


**Fixes**

- Revise HYFS to track file path

SPIFFS getFilePath() call sometimes returns weird results.

- Fix SPIFFS  FileSystem::getFilePath return code

- Fix FileDir weirdness

The generic `DirHandle` type points to an `IFS::FileDir` struct, which each filesystem defines internally as required.
It looks like the compiler sometimes mixes these up and causes problems,
though this has only been seen when compiling in debug mode for Host.

So instead, redefine `DirHandle` as an abstract type, move the `FileDir` definition inside the appropriate filesystem namespace
and add the `GET_FILEDIR()` macro to deal with checking and casting.

- Fix memory leak in test module (thanks valgrind)

- Set lasterror in `File::readContent()`

- Fix `FileSystem::makedirs()`

- Fix LFS reported volumesize


**Improvements**

- Increase LFS_CACHE_SIZE from 16 to 32 bytes

Results in signficant reduction in read counts

- Switch to littlefs fork, manage file attributes manually

Only changed attributes (and those with non-default values) get written out
File time only gets updated if file is changed, opening in write mode is insufficient

- LFS mkdir sets mtime on success

- LFS `mkdir` succeeds if directory already exists

- Add LFS inspect sample

- Implement Host `flush`, `rename` and `remove` methods

- Helpers return FileSystem* instead of IFileSystem* for easier use

- Add optional `IFileSystem::setProfiler` method to enable flexible debugging and performance evaluation

- Revise hybrid filesystem to allow use of alternative writeable filesystem (e.g. Little FS)

- Remove `IFileSystem.fopendir(Stat&)` and `IFileSystem.fopen(Stat&)`

The goal of these methods was to provide more efficient means to open files/directories discovered during enumeration.
However, they are difficult to implement effectively and the additional complexity is hard to justify.

POSIX defines a whole raft of such methods such as `openat`, `mkdirat`, etc. which add an additional `dirfd` parameter
to specific the working directory. See https://man7.org/linux/man-pages/man3/open.3p.html for some background.

This may be re-visited in the future if the need arises.

- Add generic get/set attribute methods

These support LFS file/directory attributes as well as standard attribute types via `AttributeTag`.

There is no POSIX specification for this but the linux convention (setxattr, etc.) are appropriate.
mikee47 added a commit to mikee47/Sming that referenced this pull request Feb 12, 2022
Opening on directory handle removed in SmingHub#2308
mikee47 added a commit to mikee47/Sming that referenced this pull request Feb 13, 2022
Opening on directory handle removed in SmingHub#2308
slaff pushed a commit that referenced this pull request Feb 15, 2022
This PR fixes a few problems related to handling of file/directory Access Control entries.

**IFS**

- Fix FileCopier class not copying directory attributes
- In filesystem builder, allow minifcation to fail, but print warning (to support e.g. JSON template files)

**LittleFS**

- Fix `readdir()` not setting default ACL
- Translate 'NOENT' lfs error code into 'NotFound' IFS error
- Fix missing directory attribute (imperfect fix in #2487)

Also remove unused FileStream method. Opening on directory handle removed in #2308.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants