WebJobs with Node (and node modules)

[morgs32]

I am bringing in all my NPM dependencies but it seems that makes the file paths too long.

Such as this is the error log. I even went into each node_module to bring their deeper node_modules into the topmost folder... and that didn't seem to work. I'd like to check that it's not because I'm compressing the files on a macintosh... I see some left overs in VS online indicating that azure puts node modules in a temp file and then converts them.

[01/23/2014 03:23:00 > 948219: SYS ERR ] Failed to copy job files: System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
   at System.IO.PathHelper.GetFullPathName()
   at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength)
   at System.IO.Path.GetFullPathInternal(String path)
   at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
   at System.IO.File.Copy(String sourceFileName, String destFileName, Boolean overwrite)
   at System.IO.Abstractions.FileWrapper.Copy(String sourceFileName, String destFileName, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CopyDirectoryRecursive(IFileSystem fileSystem, String sourceDirPath, String destinationDirPath, Boolean overwrite)
   at Kudu.Core.Jobs.BaseJobRunner.CacheJobBinaries(IJobLogger logger)
[01/23/2014 03:23:00 > 948219: SYS ERR ] Failed to start job due to: System.InvalidOperationException: Missing working directory
   at Kudu.Core.Jobs.BaseJobRunner.InitializeJobInstance(JobBase job, IJobLogger logger)
   at Kudu.Core.Jobs.TriggeredJobRunner.StartJobRun(TriggeredJob triggeredJob)

[davidebbo]

How long is your longest npm path? We copy it into a temp folder that looks like C:\DWASFiles\Sites\SomeSite\Temp\jobs\triggered\SomeJob\qik4k5ae.il3. So about 70 characters (depending on site & job names) are used up, leaving about 190 for your longest path within your zip.

[morgs32]

I can find one that gets to 175. But I also tried moving each sub module into the topmost node_modules folder and that didn't appear to help.... I'm interested in trying to user the WEBROOT_PATH to access modules in the root directory and not even zip or send node_modules for the job. Can you advise on how to use that environment variable in a require statement?

[davidebbo]

If the issue cannot be explained based on the size of your path, then we need to investigate why it is happening. Can you please share a repro? You can remove all your personal job files (if they're sensitive), and replace it with some dummy job. As long as it uses the same packages, it should repro.

[morgs32]

Sure! Here is the job but I removed the node_modules for size, so you will need to install. Let me know if you would rather me include them.
https://dl.dropboxusercontent.com/u/57295034/job.zip

[davidebbo]

Deepest path I see in there is:

node_modules\juice\node_modules\jsdom\node_modules\request\node_modules\form-data\node_modules\combined-stream\node_modules\delayed-stream\test\integration\test-delayed-stream-auto-pause.js

which is 190 long. So you're right at the point where you will hit this if your site and job name are each longer than 7 or 8 chars.

[morgs32]

Ok so I wrote a script to go through all modules, down the hierarchy and pull out dependencies so that I can put them in my root package.json file.
Not getting anymore errors about path length.

Now I'm getting one for disk space (I delete each job when it breaks, so I only have one loaded at a time). Zip file is 20MBs.

Looks like I have filled up the folder you told me about:
C:\DWASFiles\Sites\SomeSite\Temp\jobs\triggered\SomeJob\qik4k5ae.il3

Should I open a new issue?

[davidebbo]

We can just continue here. Can you share the repro zip for this new test case? How big are the unzipped files? Normally, there is a 200MB quota for such temp files, though some of it is likely already taken by something else.

[morgs32]

It's more or less the same except almost all the node modules are in the top folder. It's around 45MBs unzipped. Dropbox link below:
https://www.dropbox.com/s/n3g6rv66z30bp4q/new_job.zip

[01/23/2014 23:44:13 > 948219: SYS ERR ] Failed to copy job files: System.IO.IOException: There is not enough space on the disk.

[davidebbo]

When I tried I got a different error:

[01/24/2014 01:57:55] 
[01/24/2014 01:57:55] module.js:340
[01/24/2014 01:57:55]     throw err;
[01/24/2014 01:57:55]           ^
[01/24/2014 01:57:55] Error: Cannot find module './util/fetcher.js'
[01/24/2014 01:57:55]     at Function.Module._resolveFilename (module.js:338:15)
[01/24/2014 01:57:55]     at Function.Module._load (module.js:280:25)
[01/24/2014 01:57:55]     at Module.require (module.js:364:17)
[01/24/2014 01:57:55]     at require (module.js:380:17)
[01/24/2014 01:57:55]     at Object.<anonymous> (C:\DWASFiles\Sites\MyTestJobSite\Temp\jobs\triggered\BigJob\g15wbatz.llq\app.js:1:77)
[01/24/2014 01:57:55]     at Module._compile (module.js:456:26)
[01/24/2014 01:57:55]     at Object.Module._extensions..js (module.js:474:10)
[01/24/2014 01:57:55]     at Module.load (module.js:356:32)
[01/24/2014 01:57:55]     at Function.Module._load (module.js:312:12)
[01/24/2014 01:57:55]     at Function.Module.runMain (module.js:497:10)
[01/24/2014 01:57:55 > 487266: SYS ERR ] Job failed due to exit code 8

This means it successfully copied the files over, but then got into some runtime error (most likely unrelated).

My guess is that there was some random files left in TMP from your previous attempts. Could you restart the site and try again? That should cause it to get a fresh empty TMP folder.

Note that for Standard site, the TMP limit is much higher.

[davidebbo]

Please reactive if you are still seeing this and would like to move investigation forward.