[1.3.2.4] Can't have symlink to node_modules in /public

[Zodiase]

Example app to reproduce the issue: https://github.com/Zodiase/Meteor-Tests/tree/public-files-no-symlink.

This is a clean new app without any extra packages. A demo npm package and a symlink to it are added to reproduce the issue.

Issue reproduced with Meteor 1.3.2.4 on OSX El Capitan.

Expected behavior

• .meteor/local/build/programs/web.browser/app should contain static files in public.
• I should be able to access the npm package files at, for example, appurl/mdi/css/materialdesignicons.css.
• I should be able to access the static file served in public folder at appurl/foo.txt.

Actual behavior

• .meteor/local/build/programs/web.browser/app does not contain any static file in public.
• appurl/mdi/css/materialdesignicons.css brings up the app not the expected file.
• appurl/foo.txt also brings up the app not the expected file.

[zol]

I confirmed that running npm install and thus having the /public/mdi symlink point to a populated node_modules directory breaks serving /foo.txt.

[polgfred]

What is the workaround? Copy into public instead of symlink?

[Zodiase]

@polgfred Unfortunately copying is what I'm doing right now. Glad I don't have too much npm dependencies needed as public assets.

[kachkaev]

Same issue here with font-awesome – I tried to link their fonts:

cd public
ln -ls ../node_modules/font-awesome/fonts ./fonts

Font-awesome is a very popular library that provides lots of font-based icons. New icons appear as the library develops, which leads to frequent changes to the fonts files. The directory with fonts is around 1M now. If one manually copies new fonts to public every time the lib updates, the project's git repo will grow too fast. Not sure I can agree with @zol about impact:few because of this.

Until the issue is fixed, public assets from npm modules can be added to .gitignore and then copied from node_modules at each clone / deploy. That does not seem to look ideal. My current workaround:

#.gitignore
/node_modules/
/public/fonts/

# package.json
{
  "...": "...",
  "scripts": {
    "...": "...",
    "prepare-public-dir": "cp -r ./node_modules/font-awesome/fonts ./public/fonts"
  }
}

# after npm install
npm run prepare-public-dir

[ilan-schemoul]

@kachkaev interesting hack I think I'm gonna use it.
@MeteorTeam please note that I'm trying to integrate mdi node package (switching from Atmosphere because it's now recommended) and both #6846 (to integrate css, actually mdi provides scss but it's not the same for every package) and this issue seem very important and you may have want to fix that before pushing everybody to npm.

[alexsmr]

Looks like this bug was not addressed for quite a while. Having symlink in public can be useful not only to link to node_modules, but also to other external asset folders. For example, JSON schema files to be exposed to the client. Copying assets, as a workaround, - is exactly what I am trying to avoid by using symlinks.

[locales-wellington]

+1

[Lidofernandez]

Hi guys is this bug going to be fix in the next meteor releases?

The work around is actually quite ugly :(

[hwillson]

Hi all - symlinks created in the /public (or /private) directory aren't currently working because of the SymlinkLoopChecker used in /tools/isobuild/package-source.js:

meteor/tools/isobuild/package-source.js

Lines 1214 to 1217 in 8b95173

	if (loopChecker.check(dir)) {
	// pretend we found no files
	return [];
	}

The loop checker itself is being a bit overzealous, and prevents node_module symlinks because the source path of the symlink has already been included by another part of the isobuild process. So when we have a symlink in /public like

/public/fonts --> ../node_modules/font-awesome/fonts

when the isobuild process looks for asset files to include, it skips over the symlink'd files because node_modules/font-awesome/fonts was already included when the server bundle was built.

I ran out of time on this one today, but I was able to verify that commenting out the above code does allow the contents of /public symlinks to be preserved in production bundles (e.g. the font-awesome fonts are copied into the production bundle, and made available where you would expect them to be). I'll mark this a pull-requests-encouraged in case anyone here is interested in working on this. Ultimately the SymlinkLoopChecker approach should be updated to better handle potential duplicate symlinks coming from /public / /private (perhaps by excluding /public / /private from the symlink loop check altogether).

[hwillson]

Just to clarify this a bit - the SymlinkLoopChecker is first used when building an application's source, then it's used again when building application assets. When it's used to build the application source, it flags node_modules directories as being covered once. Then when it's used to build assets, the node_modules based symlinks in public / private are ignored, since they were flagged as being covered by the application source build pass. All of this happens here:

meteor/tools/isobuild/package-source.js

Lines 854 to 887 in 5e9ff34

	var sourceArch = new SourceArch(self, {
	kind: 'app',
	arch: arch,
	sourceRoot: self.sourceRoot,
	uses: uses,
	getFiles(sourceProcessorSet, watchSet) {
	sourceProcessorSet.watchSet = watchSet;
	const findOptions = {
	sourceProcessorSet,
	watchSet,
	sourceArch: this,
	ignoreFiles,
	isApp: true,
	loopChecker: new SymlinkLoopChecker(self.sourceRoot)
	};
	return {
	sources: self._findSources(findOptions).sort(
	loadOrderSort(sourceProcessorSet, arch)
	).map(relPath => {
	return {
	relPath,
	fileOptions: self._inferFileOptions(relPath, {
	arch,
	isApp: true,
	}),
	};
	}),
	assets: self._findAssets(findOptions),
	};
	}
	});

You can see that the same new SymlinkLoopChecker(self.sourceRoot) find option is used for both self._findSources and self._findAssets. I'm thinking we can probably get away with using separate SymlinkLoopChecker instances for sources and assets. This approach won't be quite as safe as using a shared checker, but it will be better then removing symlink loop checking on public / private completely.

I'll prep a PR for this.