Follow symlinks?

[jeanlauliac]

Do you want to request a feature or report a bug?

Feature

What is the current behavior?

I'm removing a test from DependencyGraph-test that looked like so, but wasn't actually working after I fixed problems with the fs mocks:

    it('should work with packages with symlinked subdirs', function() {
      var root = '/root';
      setMockFileSystem({
        'symlinkedPackage': {
          'package.json': JSON.stringify({
            name: 'aPackage',
            main: 'main.js',
          }),
          'main.js': 'lol',
          'subdir': {
            'lolynot.js': 'lolynot',
          },
        },
        'root': {
          'index.js': [
            '/**',
            ' * @providesModule index',
            ' */',
            'require("aPackage/subdir/lolynot")',
          ].join('\n'),
          'aPackage': { SYMLINK: '/symlinkedPackage' },
        },
      });

      var dgraph = new DependencyGraph({
        ...defaults,
        roots: [root],
      });
      return getOrderedDependenciesAsJSON(dgraph, '/root/index.js').then(function(deps) {
        expect(deps)
          .toEqual([
            {
              id: 'index',
              path: '/root/index.js',
              dependencies: ['aPackage/subdir/lolynot'],
              isAsset: false,
              isJSON: false,
              isPolyfill: false,
              resolution: undefined,
              resolveDependency: undefined,
            },
            {
              id: 'aPackage/subdir/lolynot.js',
              path: '/root/aPackage/subdir/lolynot.js',
              dependencies: [],
              isAsset: false,
              isJSON: false,
              isPolyfill: false,
              resolution: undefined,
              resolveDependency: undefined,
            },
          ]);
      });
    });

What is the expected behavior?

Reintroduce the test and verify symlinks work.

[lacker]

symlinks not working in React Native is one of the biggest RN complaints, so if my understanding is correct and this would fix that, this would be great!

[jeanlauliac]

I think we won't have time to deal with this anytime soon unfortunately. I'd love to reopen once we have a strong case.

[ericwooley]

Can we leave this open until it's resolved?

This is a huge PITA for code sharing, and developing modules. I'm not sure what your looking for in terms of a strong case, but it's definitely a major pain point for me, and module developers.

Currently I am trying to use lerna, and react-primitives to reorganize a project boilerplate to maximize code reuse, while maintaining upgradability by rn.

Lerna works by symlinking and installing your packages in a mono repo, which is an incredibly helpful pattern for code reuse, and completely broken when the react native packager won't follow symlinks.

[cpojer]

Yeah, it should be open. We should find a fix but we are not actively invested in making this change ourselves right now. If the community can find a reasonable solution, we should merge it and make it work.

[ericwooley]

I suspect it has to do with this line
according to the docs you may also need to check if it's a symlink. EG

//...
 this._moduleResolver = new ModuleResolver({
      dirExists: filePath => {
        try {
           var stats = fs.lstatSync(filePath);
          return  stats.isDirectory() || stats.isSymbolicLink();
        } catch (e) {}
        return false;
      },
// ...

Though I am short on time for digesting how to get started with this project at the moment. I may have more time over the weekend for a proper PR.

Or if someone else gets the chance to play around with it, that would be awesome.

I never have enough time to accomplish all the things I want to do :(

Edit: added "may", as the docs don't explicitly say you isDirectory() won't return true on a symlink, but I believe that is the case.

Edit^2: https://github.com/ericwooley/symlinkTest You do need to explicitly check on OSX Sierra at least, not sure about linux or windows.

Edit^3: the commit where the test was removed c1a8157

[haggholm]

I'd love to reopen once we have a strong case.

With npm 5, all local installs (npm i ../foo) are now symlinked. This makes it pretty rough to work with projects that have local dependencies. (That’s on top of the monorepo scenario.)

[jeanlauliac]

Edit^3: the commit where the test was removed c1a8157

The test was removed because it was already broken for a long time, unfortunately (I forgot to put that in the changeset description apparently). Now I realised I shouldn't have removed it completely, only make verify the broken behavior.

I suspect it has to do with this line

dirExists is only used for debugging, not for resolution. The resolution is done based on the HasteFS instance provided by the jest-haste-map module. I believe, but may well be wrong, that a fix for symlinks will have to be implemented in jest-haste-map, both for its watchman and plain crawlers, as well for its watchman and plain watch modes. Since, last time I've heard (might have changed!), watchman doesn't handle symlink, we were hampered in these efforts.

[ericwooley]

dirExists is only used for debugging, not for resolution. The resolution is done based on the HasteFS instance provided by the jest-haste-map module. I believe, but may well be wrong, that a fix for symlinks will have to be implemented in jest-haste-map, both for its watchman and plain crawlers, as well for its watchman and plain watch modes. Since, last time I've heard (might have changed!), watchman doesn't handle symlink, we were hampered in these efforts.

Interesting.

I am pretty confused by this project, as I am not sure how it is imported by RN. But I manually edited this change into node_modules/react-native/packager/src/node-haste/dependencyGraph.js to use my suggested code and no longer got errors about symlinked modules not being found. I got a ton of other errors, i think related to multiple installs of react-native, but I seemed to have gotten much further in the process.

I also put a console log there to offer some insight, and it was definitely logging. So that line appears to be used by the react-native project while doing real work (as opposed to debugging).

After much frustration with this issue, it ended up being easier to PR the libs i need to work with haul, rather than try to tackle this problem. I wish I had more time for this issue, but it appears to be a pretty rough one to solve, given what @jeanlauliac said.

Hopefully this gets officially resolved soon. as I would much prefer to use the built in solution.

On a side note, haul seems to have fixed all the symlink issues. Is there a downside to haul that you know of? It seems like it could be something that could be officially brought into the fold.

[bleighb]

Something that half-works is not symbolic linking but hard linking. You can't hard link directories, but you can use cp to copy all the directories and hard link every file rather than copy it with -al, e.g. cp -al <sourceDirectory> <destinationDirectory>.

The only catch is that it seems in order to trigger rebundling I have to save the changed file (even though the action of saving isn't changing the contents of the file).

[bleighb]

@ericwooley Can you explain how your PR, haul, and storybooks (?) relate to the react-native symbolic link issue?

[ericwooley]

@bleighb

Haul is an alternative to the react-native packager that doesn't have issues with symlinks. They reference this issue because those PR's and issues are about issues with symlinking and the metro bundler.

As to your cp -r issue, you might have better luck with rsync. Which is something I considered at one point.

[kschzt]

I'm in the lerna boat, too. The way I finally worked around this issue was to just explicitly add the packages in our monorepo to rn-cli.config.js, instead of having them as dependencies in the RN project's package.json (and symlinked into node_modules)

var config = {
  getProjectRoots() {
    return [
      path.resolve(__dirname), 
      path.resolve(__dirname, '../some-other-package-in-lerna-monorepo')
    ]
  }
}

May have changed for Metro. HTH

[ericwooley]

@kschzt I will have to try that solution tonight.

I imagine that there will be more pressure on this issue once yarn workspaces are released. Until then, your solution may be the only way forward.

[haggholm]

I was unable to get things to work in something like a monorepo when trying @kschzt’s approach. If I have module A as a dependency and point rn-cli.config.js to it explicitly, then it will not be found by packages requiring it from inside the ‘root’ repo. In fact, the packager doesn’t even look! The structure is probably fairly obvious from a log excerpt:

Looking for JS files in
   /mnt/scratch/shared/petter/owl-devenv/client/root/local
   /mnt/scratch/shared/petter/owl-devenv/client/root/local/@client
   /mnt/scratch/shared/petter/owl-devenv/client/root/local/@client/util
   /mnt/scratch/shared/petter/owl-devenv/client/root 

React packager ready.

Loading dependency graph, done.
warning: the transform cache was reset.
error: bundling: UnableToResolveError: Unable to resolve module `@client/util/memoizeOnce` from `/mnt/scratch/shared/petter/owl-devenv/client/root/node_modules/@client/store/bind.js`: Module does not exist in the module map or in these directories:
  /mnt/scratch/shared/petter/owl-devenv/client/root/node_modules/@client/util

Why does it not check to see if the module exists inside the local/ directory? Beats me.

(react-native v0.45.0.)

[ericwooley]

@kschzt @haggholm

How are you importing your files

1. import whatever from '../../otherPackage/someModules'?
2. import whatever from 'otherPackage/someModules'?
3. import whatever from 'someModules'?

I'm not really sure how roots work in this context, but the way your importing may be a factor. Or are you using the @providesModule syntax?

[haggholm]

@ericwooley 2(ish): import whatever from '@foo/bar/baz, in this case, with baz having .js, .android.js, and .ios.js versions for different platforms.

Both the project and I are new to React Native (migrating an existing web app); I’m not familiar with the @providesModule syntax. Perhaps I should look into that and see if it works…

[pocesar]

funny, it used to work (import { something } from 'symlinked-package'), now it doesn't anymore after upgrading react native from 0.43 to 0.45, anything changed in the packager from there?

[AshCoolman]

This lerna project seems to work with Symlinks too (react-native 0.40)

https://github.com/samcorcos/learna-react-native

[ericwooley]

I have played with that too @AshCoolman, and it does seem to work, but setting it up in a similar way with other versions has not worked.

[AshCoolman]

@ericwooley My experience too

[ktj]

Is the only solution at the moment to use haul or some custom shell scripts?

[MrLoh]

I updatet the metro-with-symlinks pacakge to create a rn-cli.config.js file that is used automatically now and updated the readme.

[sfrdmn]

BTW scoped packages don't work in extraNodeModules currently, until #173 is merged

[deusd]

@th317erd just tried your updated script, it's not including the scoped modules for me in that version. Also has to remove the line that included ./transformer to get it to work not sure if that was required. Glad the snippet was useful!

[th317erd]

Thanks @deusd ! Let me know if / how you fix it so I can update the script. Yeah... the transformer isn't supposed to be in there...

[ssavitzky]

This is very quick and dirty, but it works. I'm very new to the javascript toolchain, but I've been slinging makefiles for a long time. Goes into web/src/Makefile and app/src/Makefile; assumes that web, lib, and app are siblings.

### Makefile to get around lack of symlink support

# Rule to make subdirectories
lib/%/: ../../lib/%/
	mkdir -p $@
# Rule to copy source files from the shared library
lib/%.js: ../../lib/%.js
	cp -a $< $@

DIRS = $(addsuffix /,$(shell cd ../..; find lib -type d -print))
LIBS = $(shell cd ../..; find lib -name '*.js' -print | grep -v '[\#~]')

all:: $(DIRS)
all:: $(LIBS) | $(DIRS) # require the subdirs before copying files

.PHONY: build
build: $(LIBS) | $(DIRS)
	npm run build

You have to run make whenever you make a change in the library. If you keep forgetting, just run

while :; do sleep 10; make; done &

[mgcrea]

A simpler working rn-cli.config.js (but requires manual definition of linked modules):

const path = require('path');
const blacklist = require('metro/src/blacklist');

const LINKED_LIBS = [path.resolve(process.env.HOME, 'Developer/react-native/react-native-elements')];

module.exports = {
  extraNodeModules: {
    'react-native': path.resolve(__dirname, 'node_modules/react-native')
  },
  getProjectRoots() {
    return [
      path.resolve(__dirname),
      ...LINKED_LIBS
    ];
  },
  getBlacklistRE: function() {
    return blacklist(LINKED_LIBS.map(lib => new RegExp(`${lib}/node_modules/react-native/.*`)));
  }
};

[aleclarson]

The OP was never clear about what exactly is broken with symlinks, but I'm going to assume he's only talking about registering symlinks with the file watcher (since that's what everyone is talking about).

The following solution:

• has a minimal footprint
• does not generate a config file
• automatically locates your locally developed packages

Steps

1. npm install get-dev-paths --only=dev

2. Add the following snippet to your metro.config.js module

const fs = require('fs')
const getDevPaths = require('get-dev-paths')

const projectRoot = __dirname
module.exports = {
  // Old way
  getProjectRoots: () => Array.from(new Set(
    getDevPaths(projectRoot).map($ => fs.realpathSync($))
  )),
  // New way
  watchFolders: Array.from(new Set(
    getDevPaths(projectRoot).map($ => fs.realpathSync($))
  ))
}

Questions

• Should we consider this issue closed now?
• Should metro adopt this technique to achieve a zero-config solution?
• Are there other issues with symlinks?

[Titozzz]

i'm not really sure what is broken right now cause i'm using metro with yarn workspaces without any issue. I just have a simple custom config

[jwaldrip]

@Titozzz Can you share your config? Would love to use metro again. We currently use haul and it's considerably slower.

[mgcrea]

Yarn workspaces are working fine because every dependency is hoisted to the root of the workspace. The build issue that arises with symlinks is when you need to link an external dependency (eg. in my case react-native-elements where I had to patch things and test against my app). So the workflow is:

1. Cloning the external module locally (react-native-elements)
2. npm install; npm link
3. Linking it in your project npm link react-native-elements

The issue arise because there is conflicting multiple react-native* dependencies (the one in your project & the one in the external lib, eg. react-native-elements).

However these build issues are to be expected and do happen with webpack as well (multiple copies of react errors, etc.)

It is fixable in a not really user-friendly way with getBlacklistRE.

With webpack, it is a bit more simple with the resolve option:

config.resolve.alias['react-native-elements'] = path.join(modulesPath, 'react-native-elements');

I think the two actionable things could be:

• Improve the docs regarding symlink usage (getBlacklistRE).
• Add a resolve API like webpack.

[aleclarson]

@mgcrea Please open a new issue for that specifically, so people can more easily find the solution.

Anyone else who finds a problem with symlinks should also open a new issue, instead of posting it here. I'm in favor of locking this mega-thread so the individual problems with symlinks can be addressed separately.

[Titozzz]

@mgcrea Actually I also need the blacklist cause in one workspace a have multiple app so i'm using this to exclude any other react native than mine (I'm using no-hoist on react-native-* packages to allow multiple versions to coexist):

blacklistRE: /(nameOfRootDirectory|packages[/\\](?!nameOfCurrentPackage).*)[/\\]node_modules[/\\]react-native[/\\]/,

(Replace nameOfRootDirectory and nameOfCurrentPackage)

@jwaldrip Please DM me on twitter so we don't spam here if you need further help

@aleclarson I agree, this thread should be locked and closed, and any new issue should be treated separately with an repro example and a use case.

[jaridmargolin]

@aleclarson @Titozzz Do you mind explaining why you think this thread should be closed? As far as I can see, metro still does not support symlinks.

As @mgcrea state:

Yarn workspaces are working fine because every dependency is hoisted to the root of the workspace.

This thread is specifically about symlink support. Lerna / Yarn may individually work, but that does not mean that symlinks are working.

--

@aleclarson, I'm also not sure I understand the rationale here:

Anyone else who finds a problem with symlinks should also open a new issue, instead of posting it here.

Wouldn't it be harder to track the status of symlink support if it were spread amongst multiple issues?

--

Should metro adopt this technique to achieve a zero-config solution?

@aleclarson, if you think you solution should be built into metro, you could create PR? If metro maintainers accept, than perhaps this issue could be closed.... Until then, I personally think this thread should remain open. At the moment, It is the only method I have for being notified when/if progress is made.

[Titozzz]

Well I'm using yarn workspace with local packages that are symlinked to the root and it's working fine, so I'd like to find an example where it's not working. Why I think this thread could be closed is because it's a mix of lerna / yarn / symlinks with many different solutions to different problems, so I'm not sure if it's still relevant.

[aleclarson]

@jaridmargolin "Symlink support" is too vague, and this issue has too much stale information that newcomers have to read around.

There should be a new issue (probably created and maintained by a FB employee) where the OP states (a) which symlink-related features are missing, (b) which of those features are being worked on, and (c) what the blockers are for each missing feature. This issue would be updated when bugs are fixed or features are added.

Then, a new issue should be created for each encountered bug that's symlink-related. Each of these issues should be Github-labeled as "symlink-related" so people can easily find which bugs have been reported (via the issue search). This will keep the discussions focused on workarounds and progress updates for each specific bug.

Mega-threads are rarely the way to go. Not a good noise-to-signal ratio, IMO. You can Github-subscribe to the FB-maintained issue that reports on bug fixes and feature updates (related to symlinks), so you won't be missing out.

Of course, it's up to FB to do this, or leave the mega-thread as-is.

[aleclarson]

I've opened #257, which adds support for symlinks matching node_modules/* or node_modules/@*/*. This PR won't affect symlinks in arbitrary directories.

This means the module resolver will follow certain symlinks when necessary. PNPM works perfectly, for example!

PS: If you have symlinks in node_modules for packages you are developing, check out this comment to learn how to tell Metro to watch those packages automatically: #1 (comment)

[Swaagie]

Simple workaround using rsync (sorry Windows users): https://github.com/Swaagie/react-native-yunolink. This helped us to get unblocked for the time being. Basically provide 1..n targets to sync and let rsync do its magic. Also it will add targets to watchFolder.

Also it's nice to see some actual solutions finally getting solidified in code.

[jehartzog]

I also ended up using rsync scripts to make this work, but it was after I finished that I found wix/wml, which should work on windows and has a nice clean set of commands + saving links to a config file for a team to use.

I haven't tested it myself, but it looks like the easiest and most straightforward to accomplish this.

[Amnesthesia]

Began moving my repositories to a monorepo yesterday (MacOS), but got stuck on this. Is this still not fixed? :(