assets.json randomly munged when running gulp --production #1398
Comments
|
Sounds like race condition city. I've never personally gotten this but I'm trying to reproduce it. What does your environment look like? Also |
|
Thanks for the quick response. Unfortunately I think this is going to be hard to reproduce. Though we have seen the issue on two separate computers in the office. Not sure what all you need to know about my environment, happy to paste in anything that helps. Our gulpfile.js: https://gist.github.com/natebeaty/d8b8a5b46a9e895d1e57 (there are some svg tasks added, but we saw the same behavior after replacing with default gulpfile.js) manifest.json: |
|
For environment: What version of Node and NPM are you running? What OS are you on? Architecture (x86-32 or x86-64)? Also, is there anything else about your environment that we should probably know? Are you doing this on a slow file system, like over a network or something? The more info we have about your environment, the better, as it will help us replicate the issue so we can diagnose it. |
|
I'm on a pretty new iMac (3.4 GHz Intel Core i5) on OS X 10.10.2. Node v0.12.0, npm 2.6.1. I believe Yosemite is all x86-64, no? Node was installed via Homebrew. I have an SSD as my main drive over USB3, and am not doing any work over a network, so that shouldn't be an issue. I'm not sure if this is worth a lot of effort if no other folks are experiencing it. I just wanted to throw it out there to see if it was something obvious to look for. However, I'm certainly open to offering anything else I can do to help diagnose. |
|
@natebeaty as an aside: Your manifest.json can be rewritten as: {
"dependencies": {
"main.js": {
"files": [
"scripts/main.js"
],
"main": true
},
"main.css": {
"files": [
"styles/main.scss"
],
"main": true
},
"editor-style.css": {
"files": [
"styles/editor-style.scss"
]
},
"jquery.js": {
"bower": ["jquery"]
},
"modernizr.js": {
"bower": ["modernizr"]
},
"respond.js": {
"bower": ["respond"]
},
"svg4everybody.js": {
"bower": ["svg4everybody"]
}
},
"config": {
"devUrl": "sce.dev"
}
}The For instance if you have a bower.json that has (for example)
{
"dependencies": {
"main.js": {
"files": [
"scripts/main.js"
],
"main": true
}
}would give you a main.js file with these files concatenated together:
If you did: {
"dependencies": {
"main.js": {
"files": [
"scripts/main.js"
],
"main": true
},
"mypersonaljquery.js": {
"bower": ["jquery"]
}
}
}It would give you two files. A main.js file with these files concatenated together:
And a mypersonaljquery.js with these files
|
|
I'm not sure if my problem is related but I can't get my gulp --production to build the assets.json file. I figured it was something wrong that I'm doing because I'm new to this set up. (Also running a newer iMac with Yosemite, intalled node with package manager). My manifest.json file is original except the devurl. Any help would be greatly appreciated, I can't get my awesome theme live! |
|
@austinpray thanks for the aside, we were wondering if that was the proper way to set up manifest.json. However, when I remove the "vendor" section, I get javascript errors from History not being defined (and if I just add History.js to "vendor", then I got errors from Masonry calls). When I just add History and Masonry to the "vendor" definition, it works as expected: I assume this is from the order of inclusion? Hard to determine exactly why it's failing. This is my main.js file if it helps diagnose: https://gist.github.com/natebeaty/1384423d4ee6a778a503 |
|
@natebeaty History.js is not included without manual intervention because it has no "main" property in bower.json. https://github.com/browserstate/history.js/blob/14968aa3cf2a27335f71d26386de0a5c4073835d/bower.json Please go get this PR accepted: browserstate/history.js#384 or use overrides: https://github.com/ck86/main-bower-files#overrides-options |
|
@austinpray Ok, that makes sense. I assume Masonry has the same issue. This ended up working for me: Coming back to the original issue, we consistently see the I'm on my home machine now (2012 Retina MBP, 2.9 GHz i7, 10.10.2, Flash drive, npm 2.5.1, node v.0.12.0) and I first got this 88 byte file: But now I run it again and again and I keep getting the mangled version, unable to get a workable assets.json file that I can deploy: assets.json output: At my work machine (faster iMac), this happens about 1/3 of the time, and I just have to keep running Any ideas of other things to try? Here's the node packages on the machine: |
|
Seems crazy, but I think dashes/hyphens in my filenames may have been the culprit in munging my But... I still have |
|
@fullyint I wondered if that could be it also. I just changed the one file with hyphens (editor-style.scss) to editorStyle.scss and still had the same behavior. However, when I took out editor-style.scss entirely from manifest.json, it worked! We don't even really use that file, so this helps. It may also point to where the problem is occurring, perhaps in the styles section? Every time I add a second scss file, even if it's just test.scss with a few simple rules, it causes the mangled assets.json. |
|
I tested more after realizing I hadn't run @austinpray likes hard data... SystemOSX 10.9.5, 3.6 GHz Intel Core 2 Duo (64 bit), 8 GB RAM, SSD (low on space, feeling a bit laggy) manifest.json3 stylesheets with |
|
Following your key: |
|
Probably a race condition. After more testing, I think filenames are irrelevant and it's just a race condition, like @austinpray said. Now I think get it. Sorry. I forgot gulp's virtue of processing asychronously. Yet, I didn't find much about race conditions over at gulp-rev, so I feel some doubts. My project has nearly equal processing time for the Replicated on sage. I achieved munged @natebeaty if you're dying to test more...
@austinpray Any potential in doing a single write to
|
|
@fullyint yep definitely a race condition. Experimenting with https://github.com/OverZealous/run-sequence is giving me good results. Have to wrap up the work week today but will kill this this weekend. |
|
@fullyint I did indeed experiment with adding several more (and removing to minimal # of) scripts and styles to manifest without much change in behavior. Something odd I noticed however: if I add For now we've just removed editor-style.scss from our site since we don't need to style the WP editor, and like you, now that it's Just Working, I've gone back to getting work done. |
Prevents potential race condition where styles and scripts write to the manifest at the same time ref roots#1398
|
@natebeaty @fullyint try this out: austinpray@b254f9d Works for me |
|
Perfect, works for me! I was able to run |
|
Perfect for me too! Thank you @austinpray! If I understand, 1.4 seconds isn't a big deal for the occasional build, but if someone has a much bigger build, 6.5% slower might be nice to avoid. Of course I can't say for sure that it would be 6.5% for everybody. In my comment above, I tried to imagine an approach that keeps everything async, but runs a single Thanks again for all your work on this! |
Prevents potential race condition where styles and scripts write to the manifest at the same time ref roots#1398 Updates changelog
|
@fullyint Thank you for benchmarking! I have done some benchmarking here: https://docs.google.com/spreadsheets/d/12ZiI3QhDWGdGGHMUU6YCfVNKLncwV3qwoKbNGBQbgjc/edit?usp=sharing Setup➜ roots git:(sanicthehedgehog) gulp -v
[09:27:47] CLI version 3.8.11
[09:27:47] Local version 3.8.11
➜ roots git:(sanicthehedgehog) node -v
v0.10.36
Benchmark commandfor i in {1..10}; do time gulp --production --silent; done EnvironmentsparallelSage master before patch blockingNew patch with js and styles in sequential order TrialsBasic SageSage out of the box Heavy JSInstalled a bunch of bullshit from bower: ➜ roots git:(master) bower install --save slick.js bxslider-4 d3 lodash backbone async angular hammer.js impress.js ember angular-ui-router{
"modernizr": "2.8.2",
"jquery": "1.11.2",
"bootstrap": "3.3.4",
"slick.js": "~1.4.1",
"bxslider-4": "~4.2.3",
"d3": "~3.5.5",
"lodash": "~3.6.0",
"backbone": "~1.1.2",
"async": "~0.9.2",
"angular": "~1.3.15",
"hammer.js": "~2.0.4",
"impress.js": "~0.5.3",
"ember": "~1.11.0",
"angular-ui-router": "~0.2.13"
}
Results
So on my side I am not seeing much of a difference. Adding more stress to the build process actually lowered the perf difference between them on my side as well. RecommendationsEven worst case scenario if there was a 10% difference I would still favor the stability of the sequential manifest writes. The time when speed matters is definitely during If you want to have your cake and eat it too: you can definitely build the |
Prevents potential race condition where styles and scripts write to the manifest at the same time ref roots#1398 Updates changelog
Prevents potential race condition where styles and scripts write to the manifest at the same time ref roots#1398 Updates changelog
|
@austinpray Brilliant! Simply beautiful. Thank you so much for all your time. And for the new release! I don't expect you to look at this further, but I tested more. I had this idea that parallel processing should be faster, and I was struggling to let it go. But my results came out consistent with yours. It just emphasizes to me of the importance of getting real data to test (and disprove) my assumptions. I'll spare you the explanation why, but I wanted to test the condition in which both the JS and the CSS were super heavy (not just the JS). I also adjusted the gulpfile for a revised parallel approach in which all tasks run in parallel, but No real speed difference between sequential approach released in 8.1.1 vs my revised parallel approach: |
|
This problem still seems to occur randomly even with all mentioned fixes, having scripts randomly break do to the base configuration of a boilerplate is a rather frustrating environment to develop in. I would add more information, but everything I would mentioned is already in here. Has there been any additional progress on this repeated issue since this thread was concluded? |
|
@dskvr fwiw, I have not seen a reoccurrence of the issue since the fix, and we have deployed dozens of times over several months, and on several projects. It was very much a problem before, and has been nonexistent since. |
|
K, good to know, I'll run a diff against master to see if I can find anything. Thanks for the quick reply |
|
Also make sure your npm deps are updated. |
Just curious if anyone else has seen this. Running
gulp --productionwill sometimes output an assets.json with an incomplete or mangled file list, such as this beauty:It's maddeningly inconsistent—running
gulp --productionrepeatedly will sometimes work, then sometimes spit out a mangled file.Before I dig further, just wanted to see if anyone else has run into this before.
The text was updated successfully, but these errors were encountered: