Writing to files in loops seems to be capped at 250 file.write calls. #1

Closed
n8agrin opened this Issue Jun 3, 2009 · 1 comment

Projects

None yet

2 participants

@n8agrin
n8agrin commented Jun 3, 2009

I wrote a simple loop last night to test File I/O and noticed the following behavior. If I created a loop like so:

var times = [];
for (var i=0,j=251; i<j; i++) {
    var t1 = Date.now();
    var file = new node.fs.File();
    file.open('test.log', 'a+');
    file.write(i + ' ' + new Date() + ' Hello World!\n');
    file.close();
    times.push((Date.now() - t1));
}

'file' would contain only 250 new lines of text, while the array 'times' would show it's length to be 1000. The number of new lines added to the file seems fixed at 250. Setting the loop to 249 produced a times array with 249 items and a file with 249 new lines. Setting the loop to 251 produced a times array with 251 items and a file with 250 new lines.

My C++ is very poor, but my guess is that each event has a queue which is hard coded at 250 and my loop over flowed the queue.

Expected:
In a loop you should be able to write to files as much as you like. A web-server that is getting hit many times per second might overflow a 250 limit.

@ry
ry commented Jun 4, 2009

So, remember: this is not sequential programming. In this example what you're doing is immediately sending the operating system 250 "open()" system calls to the same file. Then you're writing to them in parallel. This is rather dangerous because it isn't guaranteed that each line is going to be printed in the right order - it could even be that characters from the various lines get mixed up with each other.

I think you probably hitting some sort of maximum number of times a single file can be open at once limit on your operating system. On Linux for example, the program handles a thousand concurrent writers.

@coleGillespie coleGillespie pushed a commit to coleGillespie/node that referenced this issue May 24, 2011
coleGillespie #include "ev/ev.h" changed to #include "ev.h" to fix the following error
bash-3.2# node-waf build
Waf: Entering directory /calipso/node_modules/node-expat/build'
[1/2] cxx: node-expat.cc -> build/default/node-expat_1.o
In file included from /usr/local/include/node/uv.h:40,
from /usr/local/include/node/node.h:25,
from ../node-expat.cc:1:
/usr/local/include/node/uv-unix.h:27:19: error: ev/ev.h: No such file or directory
In file included from /usr/local/include/node/node.h:25,
from ../node-expat.cc:1:
/usr/local/include/node/uv.h:145: error: ‘ev_timer’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_idle’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_io’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_io’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_prepare’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_check’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_idle’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_async’ does not name a type
/usr/local/include/node/uv.h:158: error: ‘ev_timer’ does not name a type
Waf: Leaving directory/calipso/node_modules/node-expat/build'
Build failed: -> task failed (err #1): 
{task: cxx node-expat.cc -> node-expat_1.o}
49528f5
@choonkeat choonkeat pushed a commit to choonkeat/node that referenced this issue Mar 28, 2012
@felixge felixge Two bug fixes for process.mixin
Bug #1 occurred when trying to use process.mixin on a function and
produced a fatal exception.

Bug #2 occurred when trying to do a deep merge with an object containing
one or more objects with a nodeType property. In those cases the deep
copy for this part of the object was not performed and a shallow one was
performed instead.

Both of these bugs were artifacts of the jQuery.extend port.
f080de5
@bnoordhuis bnoordhuis added a commit that referenced this issue Apr 10, 2013
@bnoordhuis bnoordhuis src: don't SetInternalField() in ObjectWrap dtor
Call SetPointerInInternalField(0, NULL) rather than
SetInternalField(0, Undefined()).

Fixes the following spurious NULL pointer dereference in debug builds:

  #0  0x03ad2821 in v8::internal::FixedArrayBase::length ()
  #1  0x03ad1dfc in v8::internal::FixedArray::get ()
  #2  0x03ae05dd in v8::internal::Context::global_object ()
  #3  0x03b6b87d in v8::internal::Context::builtins ()
  #4  0x03ae1871 in v8::internal::Isolate::js_builtins_object ()
  #5  0x03ab4fab in v8::CallV8HeapFunction ()
  #6  0x03ab4d4a in v8::Value::Equals ()
  #7  0x03b4f38b in CheckEqualsHelper ()
  #8  0x03ac0f4b in v8::Object::SetInternalField ()
  #9  0x06a99ddd in node::ObjectWrap::~ObjectWrap ()
  #10 0x06a8b051 in node::Buffer::~Buffer ()
  #11 0x06a8afbb in node::Buffer::~Buffer ()
  #12 0x06a8af5e in node::Buffer::~Buffer ()
  #13 0x06a9e569 in node::ObjectWrap::WeakCallback ()
cd96f0a
This was referenced Aug 29, 2013
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment