File system truncate API

[geky]

Description

From @OscarGarciaF:

How can i truncate a file to a certain size with this library?

related PelionIoT/sd-driver#99, IOTSTOR-352
cc @dannybenor, @kjbracey-arm, @ARMmbed/mbed-os-storage fyi

Issue request type

[x] Question
[ ] Enhancement
[ ] Bug

[geky]

As mentioned in PelionIoT/sd-driver#99, we do have implementations of truncate in both the ChaN FAT filesystem and LittleFS, so it should just be an API addition here:
https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/FileSystem.h#L174-L184
[Mirrored to Jira]

[OscarGarciaF]

@geky could you please help me add the API? i have no idea how to define it
I have opened FileSystem.h, FatFileSystem.h and ff.h from ChaN
i added "virtual int file_truncate(fs_file_t file);" to the first two but it does not seem to work, and i have no idea how to use this function.
i checked the documentation: and ftruncate seems to have two params while ff.h only has one
[Mirrored to Jira]

[kjbracey]

@geky I can look at doing ftruncate and FileHandle::truncate when I've got a moment. From some previous discussions, I was also fancying its opposite posix_fallocate and FileHandle::allocate.

If we do have that, does it make sense to have FileSystem::truncate also passed down to filesystem implementations, or could the be defined in terms of FileHandle::truncate?

[Mirrored to Jira]

[dannybenor]

I think we shouldn't add apis without design review
[Mirrored to Jira]

[OscarGarciaF]

Is there someway I could add it now? I need it for my work El vie., 25 de may. de 2018 9:22 AM, Daniel Benor <notifications@github.com> escribió:

I think we shouldn't add apis without design review — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7010 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AkhVKoBb8ji6Toh3wtN7I3Gog3Xtwq4Qks5t2BOkgaJpZM4UM_Wd> .

*[Mirrored to Jira]*

[geky]

Sorry I missed so many messages

@OscarGarciaF, the ChaN documentation may help:
http://elm-chan.org/fsw/ff/doc/truncate.html

It looks like truncate uses the current file position. So you will need to seek to where you want to truncate to first.

I haven't tested it, but adding this implementation to the FATFileSystem may work:

int FATFileSystem::file_truncate(fs_file_t file, off_t length)
{
    FIL *fh = static_cast<FIL*>(file);

    lock();
    // save current position
    FSIZE_T oldoff = f_tell(fh);

    // seek to new file size and truncate
    FRESULT res = f_lseek(fh, length);
    if (res) {
        unlock();
        return fat_error_remap(res);
    }

    FRESULT res = f_truncate(fh);
    if (res) {
        unlock();
        return fat_error_remap(res);
    }

    // restore old position
    FRESULT res = f_lseek(fh, oldoff);
    if (res) {
        unlock();
        return fat_error_remap(res);
    }

    return 0;
}

Unfortunately, there are several layers that the API needs to be poked through:

• Add function to FATFileSystem

  class FATFileSystem {
      virtual int file_truncate(fs_file_t file, off_t length);
  }

  https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/fat/FATFileSystem.h#L37
• Add virtual function to the abstract FileSystem class

  class FileSystem {
      virtual int file_truncate(fs_file_t file, off_t length) = 0;
  }

  https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/FileSystem.h#L49
• Add wrapper for File class

  class File {
     virtual int truncate(off_t length) {
        return _fs->file_truncate(_file, length);
     }
  }

  https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/File.h#L30

That should be enough to get you to the point where you can use the truncate API:

FATFileSystem fs("fs", &bd);

int main() {
    // with C++ api
    File f;
    f.open(&fs, "hi.txt", O_WRONLY);
    f.truncate(1000);
    f.close();
}

To add ftruncate and truncate would require toolchain specific retargetting support. This can get quite a bit more difficult.

---

If we do have that, does it make sense to have FileSystem::truncate also passed down to filesystem implementations, or could the be defined in terms of FileHandle::truncate?

@kjbracey-arm, I think retarget support to call FileHandle::truncate is sufficient, this would be poked through to the filesystems like this:

ftruncate --calls-> FileHandle::truncate --virtual-> File::truncate --calls-> FileSystem::file_truncate --virtual-> FATFileSystem::file_truncate

It's a bit of a long chain but gets you where you want and allows support at both the FileSystem and FileHandle levels.

I was also fancying its opposite posix_fallocate and FileHandle::allocate

@kjbracey-arm, Sounds fine to me, LittleFS and POSIX support allocation via truncate (counterintuitively), and ChaN has its own f_expand function. Also should be noted none of our filesystem support file holes.

I think we shouldn't add apis without design review

@dannybenor, that shouldn't stop someone from creating a PR. It's the only way for @OscarGarciaF to initiate a design review.

Fortunately ftruncate is already a standardized API as a part of POSIX:
http://pubs.opengroup.org/onlinepubs/007908799/xsh/ftruncate.html

[Mirrored to Jira]

[OscarGarciaF]

Thanks you so much @geky, you are a real life saver. I will test it out today at work El vie., 25 de may. de 2018 11:14 AM, Christopher Haster < notifications@github.com> escribió:

Sorry I missed so many messages @OscarGarciaF <https://github.com/OscarGarciaF>, the ChaN documentation may help: http://elm-chan.org/fsw/ff/doc/truncate.html It looks like truncate uses the current file position. So you will need to seek to where you want to truncate to first. I haven't tested it, but adding this implementation to the FATFileSystem may work: int FATFileSystem::file_truncate(fs_file_t file, off_t length) { FIL *fh = static_cast<FIL*>(file); lock(); // save current position FSIZE_T oldoff = f_tell(fh); // seek to new file size and truncate FRESULT res = f_lseek(fh, length); if (res) { unlock(); return fat_error_remap(res); } FRESULT res = f_truncate(fh); if (res) { unlock(); return fat_error_remap(res); } // restore old position FRESULT res = f_lseek(fh, oldoff); if (res) { unlock(); return fat_error_remap(res); } return 0; } Unfortunately, there are several layers that the API needs to be poked through: - Add function to FATFileSystem class FATFileSystem { virtual int file_truncate(fs_file_t file, off_t length); } https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/fat/FATFileSystem.h#L37 - Add virtual function to the abstract FileSystem class class FileSystem { virtual int file_truncate(fs_file_t file, off_t length) = 0; } https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/FileSystem.h#L49 - Add wrapper for File class class File { virtual int truncate(off_t length) { return _fs->file_truncate(_file, length); } } https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/File.h#L30 That should be enough to get you to the point where you can use the truncate API: FATFileSystem fs("fs", &bd); int main() { // with C++ api File f; f.open(&fs, "hi.txt", O_WRONLY); f.truncate(1000); f.close(); } To add ftruncate and truncate would require toolchain specific retargetting support. This can get quite a bit more difficult. ------------------------------ If we do have that, does it make sense to have FileSystem::truncate also passed down to filesystem implementations, or could the be defined in terms of FileHandle::truncate? @kjbracey-arm <https://github.com/kjbracey-arm>, I think retarget support to call FileHandle::truncate is sufficient, this would be poked through to the filesystems like this: ftruncate --calls-> FileHandle::truncate --virtual-> File::truncate --calls-> FileSystem::file_truncate --virtual-> FATFileSystem::file_truncate It's a bit of a long chain but gets you where you want and allows support at both the FileSystem and FileHandle levels. I was also fancying its opposite posix_fallocate and FileHandle::allocate @kjbracey-arm <https://github.com/kjbracey-arm>, Sounds fine to me, LittleFS and POSIX support allocation via truncate (counterintuitively), and ChaN has its own f_expand function. Though none of our filesystem support file holes. I think we shouldn't add apis without design review @dannybenor <https://github.com/dannybenor>, that shouldn't stop someone from creating a PR. It's the only way for @OscarGarciaF <https://github.com/OscarGarciaF> to initiate a design review. Fortunately ftruncate is already a standardized API as a part of POSIX: http://pubs.opengroup.org/onlinepubs/007908799/xsh/ftruncate.html — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7010 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AkhVKsettw6jZDIyk1Q-66pxKN11JyZQks5t2C3PgaJpZM4UM_Wd> .

*[Mirrored to Jira]*

[OscarGarciaF]

@geky im getting an error since im using FILE(aka __sFILE) instead of File
are these different classes?

[Mirrored to Jira]

[geky]

Ah yes, they are different things. Unfortunately, @kjbracey-arm correct me if I'm wrong, I don't think you can get the FileHandle from a FILE.

What you can do, is add the ftruncate function to mbed_retarget.cpp, something like this should work:

extern "C" int ftruncate(int fildes, off_t length) {
    FileHandle* fhc = get_fhc(fildes);
    int err = fhc->truncate(length);
    if (err) {
        errno = -err;
        return -1;
    }
    return 0;
}

Note this has to be in mbed_retarget.cpp, since get_fhc is internal.

Then you can use it in combination with the fileno function. I'm not sure if there's an equivalent truncate for the FILE type, but this should at least work:

int main() {
    FILE *f = fopen("/fs/hi.txt", O_WRONLY);
    int fd = fileno(f);
    ftruncate(fd, 1000);
}

[Mirrored to Jira]

[OscarGarciaF]

thanks @geky
Out of desesperation i changed my entire code to use File instead of FILE and after a few tweaks it works!
I still have no idea what the difference is between the two, why i can't use all the "f functions" with File (ftell, fopen, etc) and why there are two very similar classes with two very similar set of functions and there does not seem to be clear documentation on the mbed pages.
Should they not be unified?

[Mirrored to Jira]

[kjbracey]

@geky, no sorry, can't do fileno either - we don't have visibility of the internal of the FILE to translate FILE to anything. If you want to mix FILE and file descriptors or FileHandles you have to open the lower level yourself so you have the handle and create the FILE from it with fdopen. I'd love to fix this, but I don't have a better suggestion than "hacking" the FILE layout on a per-toolchain basis.

@OscarGarciaF - FileHandle is mbed OS' "native" API. The other APIs are the standard portable POSIX <fcntl.h>/<unistd.h> and C <stdio.h> APIs which are provided to aid compatibility. There is plenty of documentation available for those online.
[Mirrored to Jira]

[OscarGarciaF]

@kjbracey-arm if they cannot be merged then why not eliminate one to have an unified API?
[Mirrored to Jira]

[geky]

The internals of mbed-os are built on C++ classes, so at some layer (maybe internally) there will be a C++ api.

The POSIX layer is it's own feature that (in theory) lets you easily add libraries written for the desktop to your application. It's been around since mbed got off the ground and extends to other less obvious features, such as support for printf and time functions.

Maybe it would help the user experience to document the File api as the primary filesystem api for mbed-os? Then support for the POSIX api would be a happy surprise for more advanced users who are trying to support external libraries.
[Mirrored to Jira]

[adbridge]

Internal Jira reference: https://jira.arm.com/browse/IOTSTOR-446

[hierophect]

I'm sorry to ask this here but would anyone know how to implement geky's workaround for MBED 2? It seems like much the same deal, adding the virtual functions to the SDFileSystem's FATFileHandle, and then MBED's FileHandle and LocalFileSystem. However, adding the function screws with every library that uses the Stream class, which inherits FileHandle. Anybody know how to make it not make all my stream classes virtual? I've tried adding the function to Stream as well but that results in obscure linker errors.