publishing a package fails if a resource handler modifies files

[Nemo64]

I build an i18n package which uses a resource-handler to make json out of yml files. This works great, however I included some yml files for tests. When I now try to publish a new version, the bundler tries to publish the json file, not the yml file.

Here is the console output:

/Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/dev_bundle/lib/node_modules/fibers/future.js:173
                        throw(ex);
                              ^
Error: ENOENT, no such file or directory '/Users/marco/meteor/nemo64:translator/test/namespace.de_DE.json'
    at Object.fs.statSync (fs.js:689:18)
    at Object.files.copyFile (/Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/files.js:430:18)
    at /Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/package-client.js:369:11
    at Array.forEach (native)
    at Function._.each._.forEach (/Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/dev_bundle/lib/node_modules/underscore/underscore.js:79:11)
    at bundleSource (/Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/package-client.js:368:5)
    at Object.exports.publishPackage (/Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/package-client.js:651:28)
    at /Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/commands-packages.js:271:26
    at /Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/buildmessage.js:186:11
    at _.extend.withValue (/Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/fiber-helpers.js:112:14)
    at /Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/buildmessage.js:183:27
    at _.extend.withValue (/Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/fiber-helpers.js:112:14)
    at /Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/buildmessage.js:181:16
    at _.extend.withValue (/Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/fiber-helpers.js:112:14)
    at Object.capture (/Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/buildmessage.js:172:21)
    at main.registerCommand.name [as func] (/Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/commands-packages.js:268:29)
    at /Users/marco/.meteor/packages/meteor-tool/.1.0.26.p03e91++os.osx.x86_64+web.browser+web.cordova/meteor-tool-os.osx.x86_64/tools/main.js:1212:23

If my package is a too big as an example, I could create a minimal test package later.

However: my guess is that the bundler can't handle if the resource-handler changes the name of the file later, can that be?

[glasser]

Yes, please do show a minimal test package, or at the very least a step by step guide to reproducing with your maximal package (clone, check out this commit, run these commands, etc).

https://github.com/meteor/meteor/wiki/Contributing-to-Meteor#reporting-a-bug-in-meteor

Thanks!

[mizzao]

I experienced this as well, glad someone else is reporting it.

You can reproduce this by adding (locally installing) https://github.com/mizzao/meteor-jqueryui to an app, and verifying that it works. Then, try and publish it, and you will get the same error. I currently use the package as a submodule, which skips the publication process: https://github.com/mizzao/CrisisMapping.

This package uses mizzao:build-fetcher to turn .build.json files into bundled source by downloading libraries at build time. As you can see, the publisher tries to look for built files in the raw package contents. Perhaps the reason for this is that the publish code currently treats assets differently from compiled Javascript source, and anything that is not JS or CSS is an asset.

mizzao:jquery-ui$ meteor publish --create
Refreshing package metadata. This may take a moment.
Reading package...
Downloading http://code.jquery.com/ui/1.11.0/jquery-ui.min.js to jquery-ui.min.js
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/jquery-ui.min.css to jquery-ui.css
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_flat_0_aaaaaa_40x100.png to images/ui-bg_flat_0_aaaaaa_40x100.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_flat_75_ffffff_40x100.png to images/ui-bg_flat_75_ffffff_40x100.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_glass_55_fbf9ee_1x400.png to images/ui-bg_glass_55_fbf9ee_1x400.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_glass_65_ffffff_1x400.png to images/ui-bg_glass_65_ffffff_1x400.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_glass_75_dadada_1x400.png to images/ui-bg_glass_75_dadada_1x400.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_glass_75_e6e6e6_1x400.png to images/ui-bg_glass_75_e6e6e6_1x400.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_glass_95_fef1ec_1x400.png to images/ui-bg_glass_95_fef1ec_1x400.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_highlight-soft_75_cccccc_1x100.png to images/ui-bg_highlight-soft_75_cccccc_1x100.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-icons_222222_256x240.png to images/ui-icons_222222_256x240.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-icons_2e83ff_256x240.png to images/ui-icons_2e83ff_256x240.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-icons_454545_256x240.png to images/ui-icons_454545_256x240.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-icons_888888_256x240.png to images/ui-icons_888888_256x240.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-icons_cd0a0a_256x240.png to images/ui-icons_cd0a0a_256x240.png
Downloading http://code.jquery.com/ui/1.11.0/jquery-ui.min.js to jquery-ui.min.js
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/jquery-ui.min.css to jquery-ui.css
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_flat_0_aaaaaa_40x100.png to images/ui-bg_flat_0_aaaaaa_40x100.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_flat_75_ffffff_40x100.png to images/ui-bg_flat_75_ffffff_40x100.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_glass_55_fbf9ee_1x400.png to images/ui-bg_glass_55_fbf9ee_1x400.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_glass_65_ffffff_1x400.png to images/ui-bg_glass_65_ffffff_1x400.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_glass_75_dadada_1x400.png to images/ui-bg_glass_75_dadada_1x400.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_glass_75_e6e6e6_1x400.png to images/ui-bg_glass_75_e6e6e6_1x400.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_glass_95_fef1ec_1x400.png to images/ui-bg_glass_95_fef1ec_1x400.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-bg_highlight-soft_75_cccccc_1x100.png to images/ui-bg_highlight-soft_75_cccccc_1x100.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-icons_222222_256x240.png to images/ui-icons_222222_256x240.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-icons_2e83ff_256x240.png to images/ui-icons_2e83ff_256x240.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-icons_454545_256x240.png to images/ui-icons_454545_256x240.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-icons_888888_256x240.png to images/ui-icons_888888_256x240.png
Downloading http://code.jquery.com/ui/1.11.0/themes/smoothness/images/ui-icons_cd0a0a_256x240.png to images/ui-icons_cd0a0a_256x240.png
Bundling source...

/home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/dev_bundle/lib/node_modules/fibers/future.js:173
                        throw(ex);
                              ^
Error: ENOENT, no such file or directory '/home/mao/projects/CrisisMapping/mizzao:jquery-ui/images/ui-bg_flat_0_aaaaaa_40x100.png'
    at Object.fs.statSync (fs.js:689:18)
    at Object.files.copyFile (/home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/files.js:430:18)
    at /home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/package-client.js:369:11
    at Array.forEach (native)
    at Function._.each._.forEach (/home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/dev_bundle/lib/node_modules/underscore/underscore.js:79:11)
    at bundleSource (/home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/package-client.js:368:5)
    at Object.exports.publishPackage (/home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/package-client.js:651:28)
    at /home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/commands-packages.js:271:26
    at /home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/buildmessage.js:186:11
    at _.extend.withValue (/home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/fiber-helpers.js:112:14)
    at /home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/buildmessage.js:183:27
    at _.extend.withValue (/home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/fiber-helpers.js:112:14)
    at /home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/buildmessage.js:181:16
    at _.extend.withValue (/home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/fiber-helpers.js:112:14)
    at Object.capture (/home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/buildmessage.js:172:21)
    at main.registerCommand.name [as func] (/home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/commands-packages.js:268:29)
    at /home/mao/.meteor/packages/meteor-tool/.1.0.26.19y7v10++os.linux.x86_64+web.browser+web.cordova/meteor-tool-os.linux.x86_64/tools/main.js:1212:23

You will need to jump through the hoops as indicated in #2392 in order to do the publish, though.

As I mentioned in #2506, the build system seems to be getting confused by raw files and built files. In cases where files are generated as part of the build, it isn't looking in the right place for them.

[glasser]

Thanks, fixed. Should get this into 0.9.2 if not 0.9.1.

[glasser]

OK, 0.9.1 is pretty tied down, so it'll go in 0.9.2. Before then, feel free to run your publish command from git or something (or if you really want I can publish a release that fixes just this). This bug only affects publish.

[mizzao]

Yep, that's what I will do. Thanks for the quick fix.

[mizzao]

@glasser I'd like to re-publish, but looks like the commit with that fix just hangs when trying to publish:

mizzao:jquery-ui$ ~/projects/meteor/meteor publish --create
Your dependency kit is out of date. I will download the new one.
######################################################################## 100.0%
Installed dependency kit v0.3.50 in dev_bundle.

ddp: updating npm dependencies -- faye-websocket, sockjs...
Refreshing package metadata. This may take a moment.

Can't Ctrl-C out of this either, suggesting it may be a bug.

EDIT: It published successfully after forcibly killing the shell and re-running. So this bug is fixed, although there might be another one which is probably unimportant.

[glasser]

Not surprised if you saw a slow sync tonight; I moved all the package URLs to CloudFront CDN which basically meant that the whole metadata file needed to be resynced (our current sync protocol is pretty simplistic). Surprised that ctrl-c didn't work though.