Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wikijs cannot be bundled in #16

Closed
Pomax opened this issue May 20, 2016 · 31 comments
Closed

wikijs cannot be bundled in #16

Pomax opened this issue May 20, 2016 · 31 comments

Comments

@Pomax
Copy link

Pomax commented May 20, 2016

trying to add this to some code that gets bundled complains that "3:15-22 require function is used in a way in which dependencies cannot be statically extracted" meaning your code's probably doing something like this:

somevar = "...";
if (...) { somevar = "......."; }
...
var Dependency = require(somevar);

instead of this:

var Dependency = false;
if (someval===val1) Dependency = require('thing1');
else if (someval===val2) Dependency = require('thing2');
else if (someval===val2) Dependency = require('thing3');

While it might look less clean, this actually makes the code far easier to both bundle and debug, as all requirements are static, and thus traceable.

@dijs
Copy link
Owner

dijs commented May 21, 2016

I will look into this. Thanks.

@dijs
Copy link
Owner

dijs commented May 21, 2016

I am not seeing any conditional requires in my code at all... @Pomax If you can dig in and find the exact line of code it cannot compile, that would be great. In the meantime, I will work on updating the babel version in hopes that a new version will compile it without bundling issues.

@dijs
Copy link
Owner

dijs commented May 21, 2016

I updated the compiler, please try to use the master branch for your project

@Pomax
Copy link
Author

Pomax commented May 21, 2016

let me give that a try

@Pomax
Copy link
Author

Pomax commented May 21, 2016

Hm, this is very odd. If I add a package entry for master, using:

  "dependencies": {
    "arctic-redpoll": "^1.0.0",
    "cli-color": "^1.1.0",
    "inquirer": "^1.0.2",
    "md5": "^2.1.0",
    "pirates": "^2.1.1",
    "socket.io": "^1.4.5",
    "wikijs": "https://github.com/dijs/wiki/tarball/master"
  },

then a require('wikijs') yields:

C:\Users\Mike\Documents\Git projects\projects\mahjong>node
> var Wiki = require('wikijs');
Error: Cannot find module 'wikijs'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Module.require (module.js:367:17)
    at require (internal/module.js:16:19)
    at repl:1:12
    at REPLServer.defaultEval (repl.js:252:27)
    at bound (domain.js:287:14)
    at REPLServer.runBound [as eval] (domain.js:300:12)
    at REPLServer.<anonymous> (repl.js:417:12)
    at emitOne (events.js:95:20)

despite the fact that there is a node_modules/wikijs dir.

@Pomax
Copy link
Author

Pomax commented May 21, 2016

ah hang on, I know why - master does not have a dist dir. One npm run build later and the module resolves. The webpack bundling does now throw up a ton of errors though, seemingly related to the har module

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tough-cookie/lib/cookie.js
Module not found: Error: Cannot resolve module 'net' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\tough-cookie\lib
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tough-cookie/lib/cookie.js 32:10-24

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tough-cookie/package.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\tough-cookie\package.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "_args": [
|     [
|       "tough-cookie@~2.2.0",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tough-cookie/lib/cookie.js 38:14-40

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/mime-db/db.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\mime-db\db.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "application/1d-interleaved-parityfec": {
|     "source": "iana"
|   },
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/mime-db/index.js 11:17-37

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/forever-agent/index.js
Module not found: Error: Cannot resolve module 'net' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\forever-agent
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/forever-agent/index.js 6:10-24

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/forever-agent/index.js
Module not found: Error: Cannot resolve module 'tls' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\forever-agent
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/forever-agent/index.js 7:10-24

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/request/lib/har.js
Module not found: Error: Cannot resolve module 'fs' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\request\lib
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/request/lib/har.js 3:9-22

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/cache.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\cache.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "properties": {
|     "beforeRequest": {
|       "$ref": "#cacheEntry"
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 4:9-32

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/cacheEntry.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\cacheEntry.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "oneOf": [{
|     "type": "object",
|     "optional": true,
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 5:14-42

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/content.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\content.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "size",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 6:11-36

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/cookie.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\cookie.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "name",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 7:10-34

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/creator.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\creator.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "name",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 8:11-36

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/entry.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\entry.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "optional": true,
|   "required": [
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 9:9-32

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/har.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\har.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "log"
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 10:7-28

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/log.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\log.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "version",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 11:7-28

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/page.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\page.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "optional": true,
|   "required": [
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 12:8-30

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/pageTimings.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\pageTimings.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "properties": {
|     "onContentLoad": {
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 13:15-44

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/postData.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\postData.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "optional": true,
|   "required": [
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 14:12-38

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/record.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\record.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "name",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 15:10-34

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/request.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\request.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "method",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 16:11-36

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/response.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\response.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "type": "object",
|   "required": [
|     "status",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 17:12-38

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/timings.json
Module parse failed: J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\har-validator\lib\schemas\timings.json Line 2: Unexpected token :
You may need an appropriate loader to handle this file type.
| {
|   "required": [
|     "send",
|     "wait",
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/har-validator/lib/schemas/index.js 18:11-36

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tunnel-agent/index.js
Module not found: Error: Cannot resolve module 'net' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\tunnel-agent
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tunnel-agent/index.js 3:10-24

ERROR in J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tunnel-agent/index.js
Module not found: Error: Cannot resolve module 'tls' in J:\Junctions\Users\Mike\Documents\Git projects\projects\mahjong\node_modules\tunnel-agent
 @ J:/Junctions/Users/Mike/Documents/Git projects/projects/mahjong/~/tunnel-agent/index.js 4:10-24

@dijs
Copy link
Owner

dijs commented May 21, 2016

Ah right, you might have to run 'npm run build' manually in that directory
On Sat, May 21, 2016 at 11:10 AM Mike Kamermans notifications@github.com
wrote:

ah hang on, I know why - master does not have a dist dir.


You are receiving this because you commented.

Reply to this email directly or view it on GitHub
#16 (comment)

@Pomax
Copy link
Author

Pomax commented May 21, 2016

yeah, did that, which generated the dist/wikijs file, but then requiring that in leads to the error log shown.

@dijs
Copy link
Owner

dijs commented May 22, 2016

Okay. Many of those errors are webpack errors related to not having loaders for those file types.

@Pomax
Copy link
Author

Pomax commented May 22, 2016

Hmm, that may be, but without the require('wikijs') (and just that, not even anything else) there are no webpack errors, and everything bundles perfectly fine. Do you know whether the dist/wiki.js lib still relies on lots of other things that require specific webpack loaders before things'll bundle correctly?

@dijs
Copy link
Owner

dijs commented May 22, 2016

These are the dependencies:

"babel-polyfill": "^6.9.0",
"isomorphic-fetch": "^2.2.1",
"underscore": "^1.8.3"

Those should all be browser compatible...

@Pomax
Copy link
Author

Pomax commented May 23, 2016

yeah those shouldn't really be a problem... alright, I created an mcve for webpack (without babel, eslint, etc) that shows this off as minimal as I can figure.

package.json:

{
  "name": "wikitest",
  "version": "1.0.0",
  "description": "",
  "main": "test.js",
  "scripts": {
    "test": "webpack"
  },
  "author": "",
  "license": "PD",
  "dependencies": {
    "webpack": "^1.12.9",
    "wikijs": "^0.2.0"
  }
}

With a simple test.js:

var Wiki = require('wikijs');

And a pretty simple webpack config:

var webpack = require('webpack');
module.exports = {
  entry:  "./test.js",
  target: "web",
  output: {
    path: ".",
    filename: "bundle.js"
  }
};

creating these three files, running npm install and then npm test yields the same kind of errors that I was seeing in my regular project.

@dijs
Copy link
Owner

dijs commented May 23, 2016

Okay thanks for the test, I will do the same
On Sun, May 22, 2016 at 9:11 PM Mike Kamermans notifications@github.com
wrote:

yeah those shouldn't really be a problem... alright, I created an mcve for
webpack (without babel, eslint, etc) that shows this off as minimal as I
can figure.

package.json:

{
"name": "wikitest",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "webpack"
},
"author": "",
"license": "PD",
"dependencies": {
"webpack": "^1.12.9",
"wikijs": "^0.2.0"
}
}

With a simple test.js:

var Wiki = require('wikijs');

And a pretty simple webpack config:

var webpack = require('webpack');
module.exports = {
entry: "./test.js",
target: "web",
output: {
path: ".",
filename: "bundle.js"
}
};

creating these three files, running npm install and then npm test yields
the same kind of errors that I was seeing in my regular project.


You are receiving this because you commented.

Reply to this email directly or view it on GitHub
#16 (comment)

@dijs
Copy link
Owner

dijs commented May 25, 2016

I should have seen this before, the issue is not with wikijs. The issue is that you are trying to import a module that cannot run on in the browser. You will have the same issue with many other server-side only modules. I may be able to make a client-side version of this library. Would that suffice?

@Pomax
Copy link
Author

Pomax commented May 25, 2016

that would be perfect - curious though, what does wikijs rely on that that prevents browser bundling? (Most of the things it'd need to do like net-fetching, parsing, etc. have universal packages available)

@dijs
Copy link
Owner

dijs commented May 25, 2016

Good question! That is actually exactly why I just started using isomorphic-fetch.

I thought that library would be fine in both browser and node, but apparently, it is not...

I guess when webpack tried to compile it, it sees request, fs, etc... and blows up...

So, maybe we should take it up with them?? Because I would love to get this working without having to create another module...

@Pomax
Copy link
Author

Pomax commented May 25, 2016

hm, if it's https://www.npmjs.com/package/isomorphic-fetch then it looks like it's optimized for browserify. Webpack failing on system-specific APIs makes sense, but maybe if the contract for isomorphic-fetch is identical to the browser Fetch API then it might be possible to tell webpack to treat isomorphic-fetch as an external, so that it replaces require('isomorphic-fetch') with fetch when it tries to bundle.

If that works, then I guess a simple addition to the README explaining what to do when using webpack would be the entire solution. I'll see if the three example files work with that in place.

@Pomax
Copy link
Author

Pomax commented May 25, 2016

Looks like even with isomorphic-fetch externalised (which itself works without a problem), something's still trying to require('request'), which causes webpack to fail.

resulting bundle:

...
    var _parser = __webpack_require__(302);
...

/***/ },
/* 302 */
/***/ function(module, exports, __webpack_require__) {

    var utils = __webpack_require__(303);
...

/***/ },
/* 303 */
/***/ function(module, exports, __webpack_require__) {

    var request = __webpack_require__(304);

Looking at where that parser is defined in the wikijs code, it seems wiki-infobox-parser is the true culprit.

@dijs
Copy link
Owner

dijs commented May 25, 2016

Hmm.. do you think using either https://github.com/github/fetch or https://github.com/bitinn/node-fetch instead of https://github.com/matthew-andrews/isomorphic-fetch would fix the issue?

@Pomax
Copy link
Author

Pomax commented May 25, 2016

no, those won't solve the wiki-infobox-parser issue. Externalizing the fetch requirement is actually entirely possible, but that leaves the real problem of the infobox parser using require etc.

@dijs
Copy link
Owner

dijs commented May 25, 2016

OHH!!! I forgot about the parser.... Dang...

@dijs
Copy link
Owner

dijs commented May 25, 2016

I can get rid of that... My bad. I only need a small part of that library.

@dijs
Copy link
Owner

dijs commented May 25, 2016

Okay. Can you try test branch for me? #22

@Pomax
Copy link
Author

Pomax commented May 25, 2016

yup, let me test that right now

@Pomax
Copy link
Author

Pomax commented May 26, 2016

No bundling errors on the test files! Checking whether the codebase I was trying to use it in now also works...

@Pomax
Copy link
Author

Pomax commented May 26, 2016

Hm, although I do see this now:

var Wiki = require('wikijs');
var wiki = new Wiki();

leads to the following error in the browser:

utils.js:39 Uncaught TypeError: Wiki is not a constructor

(the "utils.js" seems a red herring, that's just the file that performs the console log)

@Pomax
Copy link
Author

Pomax commented May 26, 2016

ah, looks like the build system generates exports.default = Wiki;, but because webpack bundles the node code, things won't work with an (additional) module.exports = Wiki; (if I add that in, things work. At least, library wise. The wikipedia system of course generates a CORS error on regular use, but running it in a service worker should not be restricted by CORS)

@dijs
Copy link
Owner

dijs commented May 26, 2016

Excellent, so what changes do I need to make, if any?
On Wed, May 25, 2016 at 7:10 PM Mike Kamermans notifications@github.com
wrote:

ah, looks like the build system generates exports.default = Wiki;, but
because webpack bundles the node code, things won't work with an
(additional) module.exports = Wiki; (if I add that in, things work. At
least, library wise. The wikipedia system of course generates a CORS error
on regular use, but running it in a service worker should not be restricted
by CORS)


You are receiving this because you commented.

Reply to this email directly or view it on GitHub
#16 (comment)

@Pomax
Copy link
Author

Pomax commented May 26, 2016

probably just a small section in the README.md that explains that if you're using webpack, and you're experiencing problems, you probably want to use:

externals: {
  "isomorphinc-fetch": "fetch"
}

Although with the parser fix that might not even be necessary anymore (not tested that and currently not in a position to quickly test)

@dijs
Copy link
Owner

dijs commented May 26, 2016

Thanks!
On Wed, May 25, 2016 at 11:04 PM Mike Kamermans notifications@github.com
wrote:

probably just a small section in the README.md that explains that if
you're using webpack, and you're experiencing problems, you probably want
to use:

externals: {
"isometric-fetch": "fetch"
}


You are receiving this because you commented.

Reply to this email directly or view it on GitHub
#16 (comment)

@Pomax
Copy link
Author

Pomax commented May 26, 2016

and you, too!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants