Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Follow symlinks? #1

Open
jeanlauliac opened this issue Feb 27, 2017 · 207 comments · May be fixed by #257
Open

Follow symlinks? #1

jeanlauliac opened this issue Feb 27, 2017 · 207 comments · May be fixed by #257

Comments

@jeanlauliac
Copy link

@jeanlauliac jeanlauliac commented Feb 27, 2017

Do you want to request a feature or report a bug?

Feature

What is the current behavior?

I'm removing a test from DependencyGraph-test that looked like so, but wasn't actually working after I fixed problems with the fs mocks:

    it('should work with packages with symlinked subdirs', function() {
      var root = '/root';
      setMockFileSystem({
        'symlinkedPackage': {
          'package.json': JSON.stringify({
            name: 'aPackage',
            main: 'main.js',
          }),
          'main.js': 'lol',
          'subdir': {
            'lolynot.js': 'lolynot',
          },
        },
        'root': {
          'index.js': [
            '/**',
            ' * @providesModule index',
            ' */',
            'require("aPackage/subdir/lolynot")',
          ].join('\n'),
          'aPackage': { SYMLINK: '/symlinkedPackage' },
        },
      });

      var dgraph = new DependencyGraph({
        ...defaults,
        roots: [root],
      });
      return getOrderedDependenciesAsJSON(dgraph, '/root/index.js').then(function(deps) {
        expect(deps)
          .toEqual([
            {
              id: 'index',
              path: '/root/index.js',
              dependencies: ['aPackage/subdir/lolynot'],
              isAsset: false,
              isJSON: false,
              isPolyfill: false,
              resolution: undefined,
              resolveDependency: undefined,
            },
            {
              id: 'aPackage/subdir/lolynot.js',
              path: '/root/aPackage/subdir/lolynot.js',
              dependencies: [],
              isAsset: false,
              isJSON: false,
              isPolyfill: false,
              resolution: undefined,
              resolveDependency: undefined,
            },
          ]);
      });
    });

What is the expected behavior?

Reintroduce the test and verify symlinks work.

@lacker
Copy link

@lacker lacker commented Feb 27, 2017

symlinks not working in React Native is one of the biggest RN complaints, so if my understanding is correct and this would fix that, this would be great!

@jeanlauliac
Copy link
Author

@jeanlauliac jeanlauliac commented Apr 25, 2017

I think we won't have time to deal with this anytime soon unfortunately. I'd love to reopen once we have a strong case.

@ericwooley
Copy link

@ericwooley ericwooley commented Jun 15, 2017

Can we leave this open until it's resolved?

This is a huge PITA for code sharing, and developing modules. I'm not sure what your looking for in terms of a strong case, but it's definitely a major pain point for me, and module developers.

Currently I am trying to use lerna, and react-primitives to reorganize a project boilerplate to maximize code reuse, while maintaining upgradability by rn.

Lerna works by symlinking and installing your packages in a mono repo, which is an incredibly helpful pattern for code reuse, and completely broken when the react native packager won't follow symlinks.

@cpojer
Copy link
Member

@cpojer cpojer commented Jun 15, 2017

Yeah, it should be open. We should find a fix but we are not actively invested in making this change ourselves right now. If the community can find a reasonable solution, we should merge it and make it work.

@cpojer cpojer reopened this Jun 15, 2017
@ericwooley
Copy link

@ericwooley ericwooley commented Jun 15, 2017

I suspect it has to do with this line
according to the docs you may also need to check if it's a symlink. EG

//...
 this._moduleResolver = new ModuleResolver({
      dirExists: filePath => {
        try {
           var stats = fs.lstatSync(filePath);
          return  stats.isDirectory() || stats.isSymbolicLink();
        } catch (e) {}
        return false;
      },
// ...

Though I am short on time for digesting how to get started with this project at the moment. I may have more time over the weekend for a proper PR.

Or if someone else gets the chance to play around with it, that would be awesome.

I never have enough time to accomplish all the things I want to do :(

Edit: added "may", as the docs don't explicitly say you isDirectory() won't return true on a symlink, but I believe that is the case.

Edit^2: https://github.com/ericwooley/symlinkTest You do need to explicitly check on OSX Sierra at least, not sure about linux or windows.

Edit^3: the commit where the test was removed c1a8157

@haggholm
Copy link
Contributor

@haggholm haggholm commented Jun 15, 2017

I'd love to reopen once we have a strong case.

With npm 5, all local installs (npm i ../foo) are now symlinked. This makes it pretty rough to work with projects that have local dependencies. (That’s on top of the monorepo scenario.)

@jeanlauliac
Copy link
Author

@jeanlauliac jeanlauliac commented Jun 16, 2017

Edit^3: the commit where the test was removed c1a8157

The test was removed because it was already broken for a long time, unfortunately (I forgot to put that in the changeset description apparently). Now I realised I shouldn't have removed it completely, only make verify the broken behavior.

I suspect it has to do with this line

dirExists is only used for debugging, not for resolution. The resolution is done based on the HasteFS instance provided by the jest-haste-map module. I believe, but may well be wrong, that a fix for symlinks will have to be implemented in jest-haste-map, both for its watchman and plain crawlers, as well for its watchman and plain watch modes. Since, last time I've heard (might have changed!), watchman doesn't handle symlink, we were hampered in these efforts.

@ericwooley
Copy link

@ericwooley ericwooley commented Jun 16, 2017

dirExists is only used for debugging, not for resolution. The resolution is done based on the HasteFS instance provided by the jest-haste-map module. I believe, but may well be wrong, that a fix for symlinks will have to be implemented in jest-haste-map, both for its watchman and plain crawlers, as well for its watchman and plain watch modes. Since, last time I've heard (might have changed!), watchman doesn't handle symlink, we were hampered in these efforts.

Interesting.

I am pretty confused by this project, as I am not sure how it is imported by RN. But I manually edited this change into node_modules/react-native/packager/src/node-haste/dependencyGraph.js to use my suggested code and no longer got errors about symlinked modules not being found. I got a ton of other errors, i think related to multiple installs of react-native, but I seemed to have gotten much further in the process.

I also put a console log there to offer some insight, and it was definitely logging. So that line appears to be used by the react-native project while doing real work (as opposed to debugging).

After much frustration with this issue, it ended up being easier to PR the libs i need to work with haul, rather than try to tackle this problem. I wish I had more time for this issue, but it appears to be a pretty rough one to solve, given what @jeanlauliac said.

Hopefully this gets officially resolved soon. as I would much prefer to use the built in solution.

On a side note, haul seems to have fixed all the symlink issues. Is there a downside to haul that you know of? It seems like it could be something that could be officially brought into the fold.

@ghost
Copy link

@ghost ghost commented Jun 20, 2017

Something that half-works is not symbolic linking but hard linking. You can't hard link directories, but you can use cp to copy all the directories and hard link every file rather than copy it with -al, e.g. cp -al <sourceDirectory> <destinationDirectory>.

The only catch is that it seems in order to trigger rebundling I have to save the changed file (even though the action of saving isn't changing the contents of the file).

@ghost
Copy link

@ghost ghost commented Jun 20, 2017

@ericwooley Can you explain how your PR, haul, and storybooks (?) relate to the react-native symbolic link issue?

@ericwooley
Copy link

@ericwooley ericwooley commented Jun 20, 2017

@bleighb

Haul is an alternative to the react-native packager that doesn't have issues with symlinks. They reference this issue because those PR's and issues are about issues with symlinking and the metro bundler.

As to your cp -r issue, you might have better luck with rsync. Which is something I considered at one point.

@kschzt
Copy link

@kschzt kschzt commented Jun 21, 2017

I'm in the lerna boat, too. The way I finally worked around this issue was to just explicitly add the packages in our monorepo to rn-cli.config.js, instead of having them as dependencies in the RN project's package.json (and symlinked into node_modules)

var config = {
  getProjectRoots() {
    return [
      path.resolve(__dirname), 
      path.resolve(__dirname, '../some-other-package-in-lerna-monorepo')
    ]
  }
}

May have changed for Metro. HTH

@ericwooley
Copy link

@ericwooley ericwooley commented Jun 21, 2017

@kschzt I will have to try that solution tonight.

I imagine that there will be more pressure on this issue once yarn workspaces are released. Until then, your solution may be the only way forward.

@haggholm
Copy link
Contributor

@haggholm haggholm commented Jun 22, 2017

I was unable to get things to work in something like a monorepo when trying @kschzt’s approach. If I have module A as a dependency and point rn-cli.config.js to it explicitly, then it will not be found by packages requiring it from inside the ‘root’ repo. In fact, the packager doesn’t even look! The structure is probably fairly obvious from a log excerpt:

Looking for JS files in
   /mnt/scratch/shared/petter/owl-devenv/client/root/local
   /mnt/scratch/shared/petter/owl-devenv/client/root/local/@client
   /mnt/scratch/shared/petter/owl-devenv/client/root/local/@client/util
   /mnt/scratch/shared/petter/owl-devenv/client/root 

React packager ready.

Loading dependency graph, done.
warning: the transform cache was reset.
error: bundling: UnableToResolveError: Unable to resolve module `@client/util/memoizeOnce` from `/mnt/scratch/shared/petter/owl-devenv/client/root/node_modules/@client/store/bind.js`: Module does not exist in the module map or in these directories:
  /mnt/scratch/shared/petter/owl-devenv/client/root/node_modules/@client/util

Why does it not check to see if the module exists inside the local/ directory? Beats me.

(react-native v0.45.0.)

@ericwooley
Copy link

@ericwooley ericwooley commented Jun 23, 2017

@kschzt @haggholm

How are you importing your files

  1. import whatever from '../../otherPackage/someModules'?
  2. import whatever from 'otherPackage/someModules'?
  3. import whatever from 'someModules'?

I'm not really sure how roots work in this context, but the way your importing may be a factor. Or are you using the @providesModule syntax?

@haggholm
Copy link
Contributor

@haggholm haggholm commented Jun 23, 2017

@ericwooley 2(ish): import whatever from '@foo/bar/baz, in this case, with baz having .js, .android.js, and .ios.js versions for different platforms.

Both the project and I are new to React Native (migrating an existing web app); I’m not familiar with the @providesModule syntax. Perhaps I should look into that and see if it works…

@pocesar
Copy link

@pocesar pocesar commented Jun 28, 2017

funny, it used to work (import { something } from 'symlinked-package'), now it doesn't anymore after upgrading react native from 0.43 to 0.45, anything changed in the packager from there?

@AshCoolman
Copy link

@AshCoolman AshCoolman commented Jul 25, 2017

This lerna project seems to work with Symlinks too (react-native 0.40)

https://github.com/samcorcos/learna-react-native

@ericwooley
Copy link

@ericwooley ericwooley commented Jul 25, 2017

I have played with that too @AshCoolman, and it does seem to work, but setting it up in a similar way with other versions has not worked.

@AshCoolman
Copy link

@AshCoolman AshCoolman commented Jul 26, 2017

@ericwooley My experience too

@ktj
Copy link

@ktj ktj commented Jul 31, 2017

Is the only solution at the moment to use haul or some custom shell scripts?

brodybits added a commit to brodybits/react-native-module-init that referenced this issue Feb 26, 2020
brodybits added a commit to brodybits/react-native-module-init that referenced this issue Feb 26, 2020
* use path.join instead of [...].join('/') (with an extra `,` now gone)
* use JavaScript Proxy object for `resolver.extraNodeModules`
  (hopefully more dynamic), as suggested in:
  - facebook/metro#1 (comment)
* additional cleanup of comments & log message
@cristianoccazinsp
Copy link

@cristianoccazinsp cristianoccazinsp commented Apr 13, 2020

Any up to date for this? Still can't use symlinked modules.

@smourph
Copy link

@smourph smourph commented May 30, 2020

Symlinked node_modules dir is the only way to deal with iCloud sync issues (renaming node_modules to node_modules.nosync then create a symlink node_modules@ -> node_modules.nosync).

Please, fix this issue :(

@YoshiYo
Copy link

@YoshiYo YoshiYo commented Jun 2, 2020

has anybody had success yarn linking a module outside of their monorepo?

@frankenthumbs the only way were able to accomplish that was with some modifications to our config that we packagafied here: https://www.npmjs.com/package/@carimus/metro-symlinked-deps

Works like a charm, kudos!

Yeah, I used it on a yarn link and it works well 🤙🏻

@vjpr
Copy link

@vjpr vjpr commented Jun 9, 2020

When metro resolves a file it looks for the file in a jest-haste-map which also watches for changes to these files. There are two implementations for building a haste map (crawling the file tree from the project roots):

  1. Facebook's watchman native (c++, rust, etc.) library.
    jest-haste-map/build/crawlers/watchman.js
  2. Node.js implementation
    jest-haste-map/build/crawlers/node.js

Watchman

watchman doesn't support symlinks for 5 years. (facebook/watchman#105).

Node impl

The node implementation deliberately doesn't follow symlinks for reasons unknown - maybe keeping parity with watchman?

        _fs().default.lstat(file, (err, stat) => {
          activeCalls--;

          if (!err && stat && !stat.isSymbolicLink()) { <-- symlinks are just skipped
            if (stat.isDirectory()) {
              search(file);
            } else {

A simple patch below would follow symlinks - but maybe other issues would arise because symlinks can point to files outside of the project roots.

        //_fs().default.lstat(file, (err, stat) => {
        _fs().default.stat(file, (err, stat) => {
          activeCalls--;

          //if (!err && stat && !stat.isSymbolicLink()) {
          if (!err && stat) {
            if (stat.isDirectory()) {
              search(file);
            } else {
@elashpear
Copy link

@elashpear elashpear commented Jun 10, 2020

@vjpr interesting - is this much different from the following PRs?

The Jest maintainers seem to have no sense of urgency around resolving this issue and do not seem interested in any solutions that only fix the problem for metro users without more robust handling for the larger jest ecosystem.

@kesha-antonov
Copy link

@kesha-antonov kesha-antonov commented Jun 17, 2020

Guys, do you know how to use symlinked files? Not repos.

I have import file from 'path/to/symlinkedFile'

@nomi9995
Copy link

@nomi9995 nomi9995 commented Jun 19, 2020

I am making a new ecosystem where I need to run the symlink component in the project. symlink are working with create-react-app with override some properties but it is not working on expo

@vjpr
Copy link

@vjpr vjpr commented Jun 19, 2020

@nomi9995 See #1 (comment)

The package bundler for Expo/RN (metro) doesn't follow symlinks, so you have to add them manually.

@vjpr
Copy link

@vjpr vjpr commented Jun 19, 2020

@kesha-antonov Add the parent dir of the symlinkedFile to getProjectRoots.

@nomi9995
Copy link

@nomi9995 nomi9995 commented Jun 19, 2020

@nomi9995 See #1 (comment)

The package bundler for Expo/RN (metro) doesn't follow symlinks, so you have to add them manually.

rn-cli.config.js

'use strict';

const path=require("path");

var config = {
  getProjectRoots() {
    return [
      path.resolve(__dirname), 
      path.resolve(__dirname,'../../../Documents/FastTrackRN/testScreen')];
  },

  getAssetRoots() {
    return [];
  },
};

module.exports = config;

I made symlink testScreen into screen DIR and then import

import TestScreen from './screens/testScreen'

i am getting same error

Unable to resolve "./screens/testScreen" from "src/App.js"

@vjpr it is not working by manually

i am algo getting this error

● Validation Warning:

  Unknown option "getProjectRoots" with value getProjectRoots() {
    return [
      path.resolve(__dirname),
      path.resolve(__dirname,'../../../Documents/storytesting/FastTractRN/testScreen')];
  } was found.
  This is probably a typing mistake. Fixing it will remove this message.

@nomi9995
Copy link

@nomi9995 nomi9995 commented Jun 25, 2020

@rafeca @davidaurelio @mjesun @jeanlauliac @cpojer and @haggholm can you help me in

import file from 'symlinkedFile'
@nomi9995
Copy link

@nomi9995 nomi9995 commented Jun 26, 2020

any there to help me about import symlink

import file from 'symlinkedFile'
@facebook facebook locked as spam and limited conversation to collaborators Jun 26, 2020
@cpojer
Copy link
Member

@cpojer cpojer commented Jun 26, 2020

Locking this issue due to spam. Will reopen in the future.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.