_MultiPathWatcher unnecessarily watches parent folder of given path

[gorogm]

Checklist

• I have searched the existing issues for similar issues.
• I added a very descriptive title to this issue.
• I have provided sufficient information below to help reproduce this issue.

Summary

When starting streamlit in a standard way, it unexpectedly printed OSError: [Errno 24] inotify instance limit reached message, even though high number of files were not around.

_MultiPathWatcher in streamlit/watcher/event_based_path_watcher.py watches parent directories of a given path, thereby possibly including unrelated directories with files of no interest.

I believe watching for the requested directory is enough, without watching for it's parent.

Reproducible Code Example

At Streamlit startup streamlit starts watching for the important files and folders.

1. _register_necessary_watchers() in streamlit/watcher/local_sources_watcher.py calls into
2. EventBasedPathWatcher() in streamlit/watcher/event_based_path_watcher.py, which calls into
3. _MultiPathWatcher::watch_path()
4. Here, the following line steps up in the hierarchy, possibly including lot of unrelated files and folders:
folder_path = os.path.abspath(os.path.dirname(path))

Steps To Reproduce

1. Have a directory structure like this:

• main
• --code
• --data

2. Place your minimal streamlit project into code, or include something from code, so LocalSourcesWatcher will watch it
3. Place a dataset with 10000s of files into data
4. Start streamlit
5. Observe that you are out of inotify watches

Expected Behavior

Only code folder is watched for changes, which has only 10-100 code files.

Current Behavior

The parent of code: the main folder is watched for changes. With this, data folder is also watched, and as it contains 10000s of files, inotify is over the limits, and even code will not watched in reality.

Increasing the inotify limit or choosing poll policy are workarounds, but that is not the proper solution.

Is this a regression?

• Yes, this used to work in a previous version.

Debug info

• Streamlit version: 1.27
• Python version: 3.9
• Operating System: Ubuntu
• Browser: Firefox (not related)

Additional Information

No response

[github-actions]

If this issue affects you, please react with a 👍 (thumbs up emoji) to the initial post.

Your feedback helps us prioritize which bugs to investigate and address first.

[gorogm]

My idea is changing
folder_path = os.path.abspath(os.path.dirname(path))
to

        if os.path.isdir(path):
            folder_path = os.path.abspath(path)
        else:
            folder_path = os.path.abspath(os.path.dirname(path))

[watana318]

@gorogm
Hi, still having this problem? If it is not resolved, I will investigate.
The version I'm using is 1.31.0.

[benjaminkzm]

Hi @gorogm , I am trying to work on this. Could you clarify how to reproduce the error? From the above steps to reproduce, it seems a bit vague. Thank you

[senthil1288]

Hi @gorogm The interns aren't able to reproduce this issue by following the steps described above. Can you please help? How do we reach out? Are you available for a quick meet?