New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maximum buffer size for contents of readdir? #58

Closed
sarnobat opened this Issue Jul 15, 2016 · 10 comments

Comments

Projects
None yet
2 participants
@sarnobat

sarnobat commented Jul 15, 2016

I'm finding that only the first 3200 file names (approximately) added to my directory filler in readdir() get displayed in my file system, with names like (soccer\ scores\ Liverpool.txt). When I run a test filller with 9000 numerically named files (00001.txt) it's fine.

I don't know what's going on but it looks like it's buffer size related. Are there any limitations on the buffer size of the directory filler?

@EtiennePerot

This comment has been minimized.

Show comment
Hide comment
@EtiennePerot

EtiennePerot Jul 15, 2016

Owner

This isn't documented much in fuse-jna or fuse itself either. I haven't confirmed this but I would guess that it is because your code isn't checking the return value of DirectoryFiller.add. It should return true on success, false on failure (because the buffer is full). Your code should return -ErrorCodes.ENOMEM() (out of memory) in this case, such that the caller knows it needs to ask again with a bigger buffer.

Let me know if my guess is correct. If it is I'll update the documentation and examples, otherwise I'll investigate further (but for that, I need a minimal example to reproduce it).

Owner

EtiennePerot commented Jul 15, 2016

This isn't documented much in fuse-jna or fuse itself either. I haven't confirmed this but I would guess that it is because your code isn't checking the return value of DirectoryFiller.add. It should return true on success, false on failure (because the buffer is full). Your code should return -ErrorCodes.ENOMEM() (out of memory) in this case, such that the caller knows it needs to ask again with a bigger buffer.

Let me know if my guess is correct. If it is I'll update the documentation and examples, otherwise I'll investigate further (but for that, I need a minimal example to reproduce it).

@sarnobat

This comment has been minimized.

Show comment
Hide comment
@sarnobat

sarnobat Jul 17, 2016

Thanks for the quick reply. I'll post here after trying it, which may take some time.

sarnobat commented Jul 17, 2016

Thanks for the quick reply. I'll post here after trying it, which may take some time.

@sarnobat

This comment has been minimized.

Show comment
Hide comment
@sarnobat

sarnobat Aug 10, 2016

I just tried it with 30,000 files named with the following name length, and it was fine.

11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-helloworld.txt

So at this point it seems like there's something wrong in my specific code.

It's not fair for me to keep this bug open. But I may need to reopen it to get more help. Thanks for the info.

sarnobat commented Aug 10, 2016

I just tried it with 30,000 files named with the following name length, and it was fine.

11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-11111-helloworld.txt

So at this point it seems like there's something wrong in my specific code.

It's not fair for me to keep this bug open. But I may need to reopen it to get more help. Thanks for the info.

@sarnobat sarnobat closed this Aug 10, 2016

@EtiennePerot

This comment has been minimized.

Show comment
Hide comment
@EtiennePerot

EtiennePerot Aug 10, 2016

Owner

Did you add code to return ENOMEM on failure to add directory entries?

If so, leaving this open as reminder to improve documentation around this.

Owner

EtiennePerot commented Aug 10, 2016

Did you add code to return ENOMEM on failure to add directory entries?

If so, leaving this open as reminder to improve documentation around this.

@EtiennePerot EtiennePerot reopened this Aug 10, 2016

@EtiennePerot EtiennePerot self-assigned this Aug 10, 2016

@sarnobat

This comment has been minimized.

Show comment
Hide comment
@sarnobat

sarnobat Aug 11, 2016

Checking the return status of DirectoryFiller.add and returning ENOMEM if non-zero didn't have any effect.

I should be able to gather a test case. I'll provide it soon. Thanks for maintaining interesting in resolving this.

sarnobat commented Aug 11, 2016

Checking the return status of DirectoryFiller.add and returning ENOMEM if non-zero didn't have any effect.

I should be able to gather a test case. I'll provide it soon. Thanks for maintaining interesting in resolving this.

@sarnobat

This comment has been minimized.

Show comment
Hide comment
@sarnobat

sarnobat Aug 11, 2016

Reproducible Test case

Instructions

  1. Download this file, and remove the ".txt" extension
    FuseTruncationProblem.java.txt
  2. Add the fuse-jna library to your classpath
  3. Run the main method

Once the file system is mounted, then execute the following from the shell:

find -L /tmp/fusejnaproblem

You should see that only the first 3500 files get printed, even though there should be 6000 files (populated via a List using the getList() method).

Findings, Notes etc

  • My latest suspicion is that something in my specific file name data is breaking the fuse functionality (with a loop that creates numerical file names, everything works

  • It's nothing to do with buffer size afterall. A single file with this name causes issues:

    files.add("3503::Mainstream American pop culture seem completely worthless! Why aren't people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing");

Conclusion

File names can be 255 characters maximum. This is not a limitation of FUSE. NTFS and EXT4 have this maximum.

Maybe consider adding a safety check somewhere in your library so that rookies like me don't get perplexed with an otherwise fantastic library. Thanks very much for creating this.

sarnobat commented Aug 11, 2016

Reproducible Test case

Instructions

  1. Download this file, and remove the ".txt" extension
    FuseTruncationProblem.java.txt
  2. Add the fuse-jna library to your classpath
  3. Run the main method

Once the file system is mounted, then execute the following from the shell:

find -L /tmp/fusejnaproblem

You should see that only the first 3500 files get printed, even though there should be 6000 files (populated via a List using the getList() method).

Findings, Notes etc

  • My latest suspicion is that something in my specific file name data is breaking the fuse functionality (with a loop that creates numerical file names, everything works

  • It's nothing to do with buffer size afterall. A single file with this name causes issues:

    files.add("3503::Mainstream American pop culture seem completely worthless! Why aren't people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing");

Conclusion

File names can be 255 characters maximum. This is not a limitation of FUSE. NTFS and EXT4 have this maximum.

Maybe consider adding a safety check somewhere in your library so that rookies like me don't get perplexed with an otherwise fantastic library. Thanks very much for creating this.

@sarnobat sarnobat changed the title from Maximum buffer size for contents of readdir? to (resolved) Maximum buffer size for contents of readdir? Aug 11, 2016

@EtiennePerot

This comment has been minimized.

Show comment
Hide comment
@EtiennePerot

EtiennePerot Oct 30, 2016

Owner

Sorry, I didn't notice this reply until just now.

Thanks a lot for writing this testcase. However, I wasn't able to reproduce the problem with it. I ran it and the directory listing contains all ~6000 files.
Maybe this is platform-dependent? Are you on OS X? I checked this on Linux 4.6. AFAIK it has a max path limit of 4096 characters, but the filename length limit should be filesystem-dependent.

If this is an OS-X-specific problem, I could add a check on filename length that only applies on OS X.

Owner

EtiennePerot commented Oct 30, 2016

Sorry, I didn't notice this reply until just now.

Thanks a lot for writing this testcase. However, I wasn't able to reproduce the problem with it. I ran it and the directory listing contains all ~6000 files.
Maybe this is platform-dependent? Are you on OS X? I checked this on Linux 4.6. AFAIK it has a max path limit of 4096 characters, but the filename length limit should be filesystem-dependent.

If this is an OS-X-specific problem, I could add a check on filename length that only applies on OS X.

@EtiennePerot EtiennePerot changed the title from (resolved) Maximum buffer size for contents of readdir? to Maximum buffer size for contents of readdir? Oct 30, 2016

@sarnobat

This comment has been minimized.

Show comment
Hide comment
@sarnobat

sarnobat Oct 31, 2016

You can close it. It's nothing to do with buffer size. Here are the relevant remarks:

It's nothing to do with buffer size afterall. A single file with this name causes issues:
files.add("3503::Mainstream American pop culture seem completely worthless! Why aren't people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing");

Conclusion

File names can be 255 characters maximum. This is not a limitation of FUSE. NTFS and EXT4 have this maximum.

Maybe consider adding a safety check somewhere in your library so that rookies like me don't get perplexed with an otherwise fantastic library. Thanks very much for creating this.

sarnobat commented Oct 31, 2016

You can close it. It's nothing to do with buffer size. Here are the relevant remarks:

It's nothing to do with buffer size afterall. A single file with this name causes issues:
files.add("3503::Mainstream American pop culture seem completely worthless! Why aren't people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing");

Conclusion

File names can be 255 characters maximum. This is not a limitation of FUSE. NTFS and EXT4 have this maximum.

Maybe consider adding a safety check somewhere in your library so that rookies like me don't get perplexed with an otherwise fantastic library. Thanks very much for creating this.

@EtiennePerot

This comment has been minimized.

Show comment
Hide comment
@EtiennePerot

EtiennePerot Oct 31, 2016

Owner

I understood what you said, but it works for me.

With the following code, which adds only a single file under 1dir:

package net.fusejna.examples;

import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.LinkedList;
import java.util.List;

import net.fusejna.DirectoryFiller;
import net.fusejna.ErrorCodes;
import net.fusejna.FuseException;
import net.fusejna.StructFuseFileInfo.FileInfoWrapper;
import net.fusejna.StructStat.StatWrapper;
import net.fusejna.types.TypeMode.NodeType;
import net.fusejna.util.FuseFilesystemAdapterFull;

public class FuseTruncationProblem extends FuseFilesystemAdapterFull
{
    public static void main(final String... args) throws FuseException, IOException
    {
        Files.createDirectories(Paths.get("/tmp/fusejnaproblem"));
        new FuseTruncationProblem().log(false).mount("/tmp/fusejnaproblem");
    }

    final String filename = "/hello1.txt";
    final String contents = "Hello World\n";

    @Override
    public int getattr(final String path, final StatWrapper stat)
    {
        if (path.equals(File.separator)) { // Root directory
            stat.setMode(NodeType.DIRECTORY);
            return 0;
        }
        if (path.equals("/hello1.txt")) { // hello.txt
            stat.setMode(NodeType.FILE).size(contents.length());
            return 0;
        }
        if (path.endsWith("dir")) {
            stat.setMode(NodeType.DIRECTORY);
            return 0;
        }
        stat.setMode(NodeType.FILE).size(contents.length());
        return 0;
    }

    private List<String> getList()
    {
        final List<String> files = new LinkedList<String>();
        files.add(
                "3503::Mainstream American pop culture seem completely worthless! Why aren't people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing, Congress, President - Page 5 - City-Data Forum");
        return files;
    }

    @Override
    public int read(final String path, final ByteBuffer buffer, final long size, final long offset, final FileInfoWrapper info)
    {
        // Compute substring that we are being asked to read
        final String s = contents.substring((int) offset,
                (int) Math.max(offset, Math.min(contents.length() - offset, offset + size)));
        buffer.put(s.getBytes());
        return s.getBytes().length;
    }

    @Override
    public int readdir(final String path, final DirectoryFiller filler)
    {
        filler.add(filename);
        if (path.length() == 1) {
            filler.add("1dir");
        }
        if (path.endsWith("1dir")) {
            for (final String file : getList()) {
                final boolean added = filler.add(file);
                if (!added) {
                    System.out.println("HelloWorldFuse.readdir() Failed to add");
                    return ErrorCodes.ENOMEM();
                }
            }
        }
        return 0;
    }
}

I get:

$ ls /tmp/fusejnaproblem/1dir    
'3503::Mainstream American pop culture seem completely worthless! Why aren'\''t people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing, Congress, President - Page 5 - City-Data Forum'
hello1.txt

Hence me asking if the behaviour is platform-specific.

Owner

EtiennePerot commented Oct 31, 2016

I understood what you said, but it works for me.

With the following code, which adds only a single file under 1dir:

package net.fusejna.examples;

import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.LinkedList;
import java.util.List;

import net.fusejna.DirectoryFiller;
import net.fusejna.ErrorCodes;
import net.fusejna.FuseException;
import net.fusejna.StructFuseFileInfo.FileInfoWrapper;
import net.fusejna.StructStat.StatWrapper;
import net.fusejna.types.TypeMode.NodeType;
import net.fusejna.util.FuseFilesystemAdapterFull;

public class FuseTruncationProblem extends FuseFilesystemAdapterFull
{
    public static void main(final String... args) throws FuseException, IOException
    {
        Files.createDirectories(Paths.get("/tmp/fusejnaproblem"));
        new FuseTruncationProblem().log(false).mount("/tmp/fusejnaproblem");
    }

    final String filename = "/hello1.txt";
    final String contents = "Hello World\n";

    @Override
    public int getattr(final String path, final StatWrapper stat)
    {
        if (path.equals(File.separator)) { // Root directory
            stat.setMode(NodeType.DIRECTORY);
            return 0;
        }
        if (path.equals("/hello1.txt")) { // hello.txt
            stat.setMode(NodeType.FILE).size(contents.length());
            return 0;
        }
        if (path.endsWith("dir")) {
            stat.setMode(NodeType.DIRECTORY);
            return 0;
        }
        stat.setMode(NodeType.FILE).size(contents.length());
        return 0;
    }

    private List<String> getList()
    {
        final List<String> files = new LinkedList<String>();
        files.add(
                "3503::Mainstream American pop culture seem completely worthless! Why aren't people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing, Congress, President - Page 5 - City-Data Forum");
        return files;
    }

    @Override
    public int read(final String path, final ByteBuffer buffer, final long size, final long offset, final FileInfoWrapper info)
    {
        // Compute substring that we are being asked to read
        final String s = contents.substring((int) offset,
                (int) Math.max(offset, Math.min(contents.length() - offset, offset + size)));
        buffer.put(s.getBytes());
        return s.getBytes().length;
    }

    @Override
    public int readdir(final String path, final DirectoryFiller filler)
    {
        filler.add(filename);
        if (path.length() == 1) {
            filler.add("1dir");
        }
        if (path.endsWith("1dir")) {
            for (final String file : getList()) {
                final boolean added = filler.add(file);
                if (!added) {
                    System.out.println("HelloWorldFuse.readdir() Failed to add");
                    return ErrorCodes.ENOMEM();
                }
            }
        }
        return 0;
    }
}

I get:

$ ls /tmp/fusejnaproblem/1dir    
'3503::Mainstream American pop culture seem completely worthless! Why aren'\''t people interested in more worthwhile activies? - Politics and Other Controversies -Democrats, Republicans, Libertarians, Conservatives, Liberals, Third Parties, Left-Wing, Right-Wing, Congress, President - Page 5 - City-Data Forum'
hello1.txt

Hence me asking if the behaviour is platform-specific.

@sarnobat

This comment has been minimized.

Show comment
Hide comment
@sarnobat

sarnobat Oct 31, 2016

Oh I see, my mistake. I was running this on Mac OS 10.7 I believe.

sarnobat commented Oct 31, 2016

Oh I see, my mistake. I was running this on Mac OS 10.7 I believe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment