Skip to content
This repository

IO event flaw - no 'user objects' on IO operations in NodeJS? #5185

Closed
jsfunfun opened this Issue · 3 comments

3 participants

jsfunfun Nathan Rajlich Isaac Z. Schlueter
jsfunfun

Hi, I've been testing NodeJS for 6 months now. An overall objective of NodeJS appears to be event IO, but no 'user objects' are returned in callbacks. Would someone care to suggest how to best support user-objects in the event model? Do I misunderstand something about NodeJS?

Thank you for your consideration and thoughts. The following demo example (copy files across a network) illustrates my point.

Issue: There is no way to track WHICH INSTANCE of an event callback is being fired.

This simple example demonstrates the flaw. This can happen on ANY IO event. In the case of fs.readFile, the callbacks may and do fire in any random order - which is correct.

However, the issue is that one does not know WHICH 'particular' event fired the callback. This is why 'user objects' are traditionally passed around in event-driven architectures.

Isn't NodeJS missing a crucial point of event-driven IO?

////
//     Example of NodeJS event IO flaw
////

var fileList = ['testOne.txt', 'testTwo.txt', 'testThree.txt'];
var operationList = [];
var nPendingOps = 0;

for (var i = 0, IL = fileList.length; i < IL; ++i)
{
    var uo = {   'rdName'  : 'f:/OldNetworkFolder/' + fileList[i]
               , 'wrName'  : 'h:/NewNetworkFolder/' + fileList[i]
               , 'fSize'   : 0
             };

    // place it on our pending operation list...
    operationList.push(uo);

    // read the old file
    ++nPendingOps;
    fs.readFile(uo.rdName, function (err, rawData) 
    {
        if (err) {/*omitted*/}

        // track/update the file size
        uo.fSize = rawData.length;

        // write the new file
        fs.writeFile(uo.wrName, rawData, function(err)
        {
            if (err) {/*omitted*/}

            if (--nPendingOps === 0)
            {
                // we've performed and completed all operations!
                onMasterCallBack('we are done.');
            }
        });
    });
} // for each file in list

What's the issue?

The contents of the WRONG file are written to the target output file during the fs.writeFile call.

The value of 'uo' is set when the readFile callback function is created. However, the value of 'uo' at the time writeFile is called is not reliable.

The error, and behavior of what I see happening is that when fs.writeFile is executed, the value of 'uo' is NOT the value when set during the anonymous function creation, but rather it has the value of when the callback (or some other moment in time) is created, or other loop iteration.

Can someone explain why 'uo' is not the correct value when fs.writeFile is called?

The fact that IO operations do not accept nor return user objects makes NodeJS unreliable. It forces programmers to cook-up tracking dictionaries for every IO operation that is called. Is this a nightmare?

Every IO operation needs to both accept a User Object AND return the User Object. For example:

fs.readFile(myObject, fileName, function (err, rawData, myObjectReturned)

Yes, there is a workable but hard to manage solution. One has to create 'local' instance versions of the user objects inside of the callback function.

BUT - the performance downside of this is large. This forces the parsing, compiling, and creation of anonymous functions for what could/would normally be defined as a more static function.

The above fs.readFile ... fs.writeFile could be fixed by adding:

fs.readFile(uo.rdName, function (err, rawData) 
{
    // create a ptr to the outer user object
    var localUO = uo;
    .
    .
    .

    // track/update the file size
    localUO.fSize = rawData.length;

    // write the new file
    fs.writeFile(localUO.wrName, rawData, function(err)
    .
    .
    .
});

The trouble with this approach is that now, because we had to create a new ptr to a local variable - that variable/ptr is NOT the same item that is in the outer scope. Therefore it's NOT the same item as our 'pending' operation's list 'operationList'.

Ultimately, now to do this one has to build custom UUID lookup maps to 'glue things together.'

What is going on with the outer level user-object? Is there a better or recommended approach to User Objects and event IO?

Nathan Rajlich
Collaborator

libuv, the IO lib that node wraps, has an API like what you're wanting (passing around "user objects"). This is a very necessary feature in C since there's no other way to get contextual information about an async operation, much like what you're wanting.

The reason node doesn't have these "user objects" for every async operation is because it's not necessary, since JavaScript has closures. That is, variables declared outside your callback function will still be available for access within your callback function. i.e.

var obj = { filename: 'foo.txt', other: 'properties' };
fs.readFile(obj.filename, function(err, data) {
  // we can still access `obj` from here, so you already have your "user object"
});
Nathan Rajlich
Collaborator

As for your first example, the reason the uo variable is not the one you're expecting is because you're defining a global uo variable, and then reassigning it 3 times in a row in a loop before the callbacks have a chance to get the expected values.

What you want to do there is to use fileList.forEach() rather than a for-loop. This will make each loop iteration be run inside of a new closure, so the defined uo variable won't get trampled on by the next iteration.

Isaac Z. Schlueter
Collaborator

@jsfunfun What @TooTallNate is essentially saying is that this is a feature/bug in the javascript language. Consider this:

for (var i = 0; i < 10; i ++) {
  setTimeout(function() {
    console.log('timeout %d', i);
  });
}

Because functions close over variables, rather than values, every time i is updated by the for loop, all 10 timeout functions see the new value when they reference the i variable, and it outputs timeout 10 every time.

One common approach is to close over the specific value by assigning it to a new variable inside an IIFE (immediately invoked function expression). Consider this:

for (var i = 0; i < 10; i ++) {
  ;(function(iLocal) {
    setTimeout(function() {
      console.log('timeout %d', iLocal);
    });
  })(i);
}

This will pass the i value and bind it in the function-local parameter iLocal. So, even if i then changes, iLocal will be created new for each iteration.

Since you are iterating over an array, you can get around this by replacing your for loop with the forEach method.

var list = [0, 1, 2];
list.forEach(function(i) {
  setTimeout(function() {
    console.log('timeout %d', i);
  });
});

In this case (much like the IIFE-in-a-loop example above), the value is bound to a local argument, so it won't change with each iteration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.