module importing performance regression

[daveroundy]

BPO	23916
Nosy	@brettcannon, @ncoghlan, @pitrou, @ericsnowcurrently
Files	test.py: script to demonstrate performance regression.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-04-12.00:34:21.594>
created_at = <Date 2015-04-11.20:07:34.862>
labels = ['performance']
title = 'module importing performance regression'
updated_at = <Date 2017-07-16.21:19:52.627>
user = 'https://bugs.python.org/daveroundy'

bugs.python.org fields:

activity = <Date 2017-07-16.21:19:52.627>
actor = 'brett.cannon'
assignee = 'none'
closed = True
closed_date = <Date 2015-04-12.00:34:21.594>
closer = 'pitrou'
components = []
creation = <Date 2015-04-11.20:07:34.862>
creator = 'daveroundy'
dependencies = []
files = ['47016']
hgrepos = []
issue_num = 23916
keywords = []
message_count = 10.0
messages = ['240491', '240492', '240493', '240494', '240500', '240514', '240515', '298433', '298434', '298450']
nosy_count = 5.0
nosy_names = ['brett.cannon', 'ncoghlan', 'pitrou', 'eric.snow', 'daveroundy']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue23916'
versions = ['Python 3.4', 'Python 3.5']

[daveroundy]

I have observed a performance regression in module importing. In python 3.4.2, importing a module from the current directory (where the script is located) causes the entire directory to be read. When there are many files in this directory, this can cause the script to run very slowly.

In python 2.7.9, this behavior is not present.

It would be preferable (in my opinion) to revert the change that causes python to read the entire user directory.

[pitrou]

This change is actually an optimization. The directory is only read once and its contents are then cached, which allows for much quicker imports when multiple modules are in the directory (common case of a Python package).

Can you tell us more about your setup?

• how many files are in the directory
• what filesystem is used
• whether the filesystem is local or remote (e.g. network-attached)
• your OS and OS version

Also, how long is "very slowly"?

[daveroundy]

I had suspected that might be the case. At this point mostly it's just a
test case where I generated a lot of files to demonstrate the issue. In my
test case hello world with one module import takes a minute and 40 seconds.
I could make it take longer, of course, by creating more files.

I do think scaling should be a consideration when introducing
optimizations, when if getdents is usually pretty fast. If the script
directory is normally the last one in the search path couldn't you skip the
listing of that directory without losing your optimization?

On Sat, Apr 11, 2015, 1:37 PM Antoine Pitrou <report@bugs.python.org> wrote:

Antoine Pitrou added the comment:

This change is actually an optimization. The directory is only read once
and its contents are then cached, which allows for much quicker imports
when multiple modules are in the directory (common case of a Python
package).

Can you tell us more about your setup?

• how many files are in the directory
• what filesystem is used
• whether the filesystem is local or remote (e.g. network-attached)
• your OS and OS version

Also, how long is "very slowly"?

----------
nosy: +pitrou

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue23916\>

---

[pitrou]

I was asking questions because I wanted to have more precise data. I can't reproduce here: even with 500000 files in a directory, the first import takes 0.2s, not one minute.

[pitrou]

As for your question:

If the script
directory is normally the last one in the search path couldn't you
skip the
listing of that directory without losing your optimization?

Given the way the code is architected, that would complicate things significantly. Also it would introduce a rather unexpected discrepancy.

[daveroundy]

My tests involved 8 million files on an ext4 file system. I expect that
accounts for the difference. It's true that it's an excessive number of
files, and maybe the best option is to ignore the problem.

On Sat, Apr 11, 2015 at 2:52 PM Antoine Pitrou <report@bugs.python.org>
wrote:

Antoine Pitrou added the comment:

As for your question:

> If the script
> directory is normally the last one in the search path couldn't you
> skip the
> listing of that directory without losing your optimization?

Given the way the code is architected, that would complicate things
significantly. Also it would introduce a rather unexpected discrepancy.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue23916\>

---

[pitrou]

Indeed, that doesn't sound like something we want to support. I'm closing then.

[daveroundy]

Here is a little script to demonstrate the regression (which yes, is still bothering me).

[pitrou]

Thanks for the reproducer. I haven't changed my mind on the resolution, as it is an extremely unlikely usecase (a directory with 1e8 files is painful to manage with standard command-line tools). I suggest you change your approach, for example you could use a directory hashing scheme to spread the files into smaller subdirectories.

[brettcannon]

I agree with Antoine that this shouldn't change. Having said that, it
wouldn't be hard to write your own finder using importlib that doesn't get
the directory contents and instead checks for the file directly (and you
could even set it just for your troublesome directory to get the
performance benefit from the default finder).

On Sun, Jul 16, 2017, 05:25 Antoine Pitrou, <report@bugs.python.org> wrote:

Antoine Pitrou added the comment:

Thanks for the reproducer. I haven't changed my mind on the resolution,
as it is an extremely unlikely usecase (a directory with 1e8 files is
painful to manage with standard command-line tools). I suggest you change
your approach, for example you could use a directory hashing scheme to
spread the files into smaller subdirectories.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue23916\>

---