New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache non-symlinks in realpathSync #10253

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet
8 participants
@yallop
Contributor

yallop commented Dec 13, 2016

Background

The fs.realpathSync function synchronously traverses paths componentwise to resolve symbolic links.

The optional cache argument stores the results of resolution to avoid repeated lookups. The cache can be used to override resolution and to store the results of calls to readlinkSync.

However, the cache only records paths that lstat reveals to be symlinks. Consequently, a call to fs.realpathSync for a path without symlinks may result in many calls to lstat, even if most parts of the path have been resolved in an earlier call.

Checklist
  • make -j4 test (UNIX), or vcbuild test nosign (Windows) passes
  • commit message follows commit guidelines
Affected core subsystem(s)

fs

Description of change

This tiny patch extends fs.realpathSync to record the results for paths that are not symlinks. The code suggests that this was originally the intention, since the cache is expected to contain entries that resolve to themselves:

cache.get(base) === base

With this patch a run of ember build on a fresh application makes around 6,200 fewer lstat calls (out of a total of around 70,000 syscalls). On a typical native file system the performance difference is negligible, but in situations where a syscall has more significant overhead it can be a worthwhile saving. For example, with a file system that forwards operations over a socket via FUSE this patch saves around 10% of the build time.

Cache non-symlinks in realpathSync.
Extend `fs.realpathSync` to cache the results for paths that are not
symlinks in addition to caching symlink mappings.

Signed-off-by: Jeremy Yallop <yallop@docker.com>
@mscdex

This comment has been minimized.

Show comment
Hide comment
@mscdex

mscdex Dec 13, 2016

Contributor

What about fs.realPath()?

Contributor

mscdex commented Dec 13, 2016

What about fs.realPath()?

@addaleax

This comment has been minimized.

Show comment
Hide comment
@addaleax

addaleax Dec 13, 2016

Member

What about fs.realpath()?

We removed the cache argument in v6. The cache in fs.realpathSync is exlusively used by the module loader now, not part of a public API.

Member

addaleax commented Dec 13, 2016

What about fs.realpath()?

We removed the cache argument in v6. The cache in fs.realpathSync is exlusively used by the module loader now, not part of a public API.

@yallop

This comment has been minimized.

Show comment
Hide comment
@yallop

yallop Dec 14, 2016

Contributor

Thanks for the reviews. I'm not sure why the test/arm CI failed, but let me know if there's anything more I need to do here.

Contributor

yallop commented Dec 14, 2016

Thanks for the reviews. I'm not sure why the test/arm CI failed, but let me know if there's anything more I need to do here.

@gibfahn

This comment has been minimized.

Show comment
Hide comment
Member

gibfahn commented Dec 23, 2016

@Trott

This comment has been minimized.

Show comment
Hide comment
@Trott

Trott Dec 23, 2016

Member

OS X failure looks like an unrelated race condition in test-http-client-timeout-with-data to me but let's re-run CI to confirm...

CI: https://ci.nodejs.org/job/node-test-pull-request/5561/

Member

Trott commented Dec 23, 2016

OS X failure looks like an unrelated race condition in test-http-client-timeout-with-data to me but let's re-run CI to confirm...

CI: https://ci.nodejs.org/job/node-test-pull-request/5561/

@Trott

This comment has been minimized.

Show comment
Hide comment
@Trott

Trott Dec 23, 2016

Member

Hmmm, now an AIX failure, also seemingly unrelated. Peculiar.

Let's try again:

CI: https://ci.nodejs.org/job/node-test-pull-request/5562/

Member

Trott commented Dec 23, 2016

Hmmm, now an AIX failure, also seemingly unrelated. Peculiar.

Let's try again:

CI: https://ci.nodejs.org/job/node-test-pull-request/5562/

@Trott

This comment has been minimized.

Show comment
Hide comment
@Trott

Trott Dec 23, 2016

Member

There we go. CI is

Member

Trott commented Dec 23, 2016

There we go. CI is

jasnell added a commit that referenced this pull request Dec 24, 2016

fs: cache non-symlinks in realpathSync.
Extend `fs.realpathSync` to cache the results for paths that are not
symlinks in addition to caching symlink mappings.

PR-URL: #10253
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
@jasnell

This comment has been minimized.

Show comment
Hide comment
@jasnell

jasnell Dec 24, 2016

Member

Landed in 5dc4487

Member

jasnell commented Dec 24, 2016

Landed in 5dc4487

@jasnell jasnell closed this Dec 24, 2016

@yallop yallop deleted the yallop:cache-non-symlinks branch Dec 26, 2016

@evanlucas evanlucas referenced this pull request Jan 3, 2017

Merged

v7.4.0 release proposal #10589

evanlucas added a commit that referenced this pull request Jan 3, 2017

fs: cache non-symlinks in realpathSync.
Extend `fs.realpathSync` to cache the results for paths that are not
symlinks in addition to caching symlink mappings.

PR-URL: #10253
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>

evanlucas added a commit that referenced this pull request Jan 4, 2017

fs: cache non-symlinks in realpathSync.
Extend `fs.realpathSync` to cache the results for paths that are not
symlinks in addition to caching symlink mappings.

PR-URL: #10253
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
@MylesBorins

This comment has been minimized.

Show comment
Hide comment
@MylesBorins

MylesBorins Jan 23, 2017

Member

I've gone ahead and backported to v6.x, but am a little bit wary. Is this a good change to backport? It did not land cleanly on v4.x, which makes sense to me. As such I added the appropriate don't land label

Member

MylesBorins commented Jan 23, 2017

I've gone ahead and backported to v6.x, but am a little bit wary. Is this a good change to backport? It did not land cleanly on v4.x, which makes sense to me. As such I added the appropriate don't land label

MylesBorins added a commit that referenced this pull request Jan 23, 2017

fs: cache non-symlinks in realpathSync.
Extend `fs.realpathSync` to cache the results for paths that are not
symlinks in addition to caching symlink mappings.

PR-URL: #10253
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>

MylesBorins added a commit that referenced this pull request Jan 24, 2017

fs: cache non-symlinks in realpathSync.
Extend `fs.realpathSync` to cache the results for paths that are not
symlinks in addition to caching symlink mappings.

PR-URL: #10253
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>

@MylesBorins MylesBorins referenced this pull request Jan 24, 2017

Merged

v6.10.0 proposal #10974

MylesBorins added a commit that referenced this pull request Jan 31, 2017

fs: cache non-symlinks in realpathSync.
Extend `fs.realpathSync` to cache the results for paths that are not
symlinks in addition to caching symlink mappings.

PR-URL: #10253
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>

MylesBorins added a commit that referenced this pull request Feb 21, 2017

2017-02-21, Version 6.10.0 'Boron' (LTS)
Notable Changes:

The SEMVER-MINOR changes include:

*  crypto: allow adding extra certs to well-known CAs (Sam Roberts)
   #9139
*  deps: Upgrade INTL ICU to version 58 (Steven R. Loomis)
   #9234
*  process: add `process.memoryUsage.external` (Fedor Indutny)
   #9587
*  src: add wrapper for process.emitWarning() (Sam Roberts)
   #9139

Notable SEMVER-PATCH changes include:

*  fs: cache non-symlinks in realpathSync. (Jeremy Yallop)
   #10253
*  repl: allow autocompletion for scoped packages (Evan Lucas)
   #10296

MylesBorins added a commit that referenced this pull request Feb 22, 2017

2017-02-21, Version 6.10.0 'Boron' (LTS)
Notable Changes:

The SEMVER-MINOR changes include:

*  crypto: allow adding extra certs to well-known CAs (Sam Roberts)
   #9139
*  deps: Upgrade INTL ICU to version 58 (Steven R. Loomis)
   #9234
*  process: add `process.memoryUsage.external` (Fedor Indutny)
   #9587
*  src: add wrapper for process.emitWarning() (Sam Roberts)
   #9139

Notable SEMVER-PATCH changes include:

*  fs: cache non-symlinks in realpathSync. (Jeremy Yallop)
   #10253
*  repl: allow autocompletion for scoped packages (Evan Lucas)
   #10296

PR-URL: #10974

imyller added a commit to imyller/meta-nodejs that referenced this pull request Mar 2, 2017

2017-02-21, Version 6.10.0 'Boron' (LTS)
    Notable Changes:

    The SEMVER-MINOR changes include:

    *  crypto: allow adding extra certs to well-known CAs (Sam Roberts)
       nodejs/node#9139
    *  deps: Upgrade INTL ICU to version 58 (Steven R. Loomis)
       nodejs/node#9234
    *  process: add `process.memoryUsage.external` (Fedor Indutny)
       nodejs/node#9587
    *  src: add wrapper for process.emitWarning() (Sam Roberts)
       nodejs/node#9139

    Notable SEMVER-PATCH changes include:

    *  fs: cache non-symlinks in realpathSync. (Jeremy Yallop)
       nodejs/node#10253
    *  repl: allow autocompletion for scoped packages (Evan Lucas)
       nodejs/node#10296

Signed-off-by: Ilkka Myller <ilkka.myller@nodefield.com>

imyller added a commit to imyller/meta-nodejs that referenced this pull request Mar 2, 2017

2017-02-21, Version 6.10.0 'Boron' (LTS)
    Notable Changes:

    The SEMVER-MINOR changes include:

    *  crypto: allow adding extra certs to well-known CAs (Sam Roberts)
       nodejs/node#9139
    *  deps: Upgrade INTL ICU to version 58 (Steven R. Loomis)
       nodejs/node#9234
    *  process: add `process.memoryUsage.external` (Fedor Indutny)
       nodejs/node#9587
    *  src: add wrapper for process.emitWarning() (Sam Roberts)
       nodejs/node#9139

    Notable SEMVER-PATCH changes include:

    *  fs: cache non-symlinks in realpathSync. (Jeremy Yallop)
       nodejs/node#10253
    *  repl: allow autocompletion for scoped packages (Evan Lucas)
       nodejs/node#10296

Signed-off-by: Ilkka Myller <ilkka.myller@nodefield.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment