Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SaveAsSync? #62

Open
kpetrow opened this issue May 5, 2017 · 2 comments
Open

SaveAsSync? #62

kpetrow opened this issue May 5, 2017 · 2 comments

Comments

@kpetrow
Copy link

kpetrow commented May 5, 2017

I needed to implement a SaveAsSync due to using a jbin in a module that couldn't work well with a call back. I can add a PR to implement a SaveAsSync if any interest.

return proto.saveAsSync = promising(function(dest, mimeType, callback) {
       if ("string" == typeof dest) {
           var buffer = this.read("blob", 0);
           is(buffer, Buffer) || (buffer = new Buffer(buffer)), require("fs").writeFileSync(dest, buffer);
       } else callback(new TypeError("Unsupported storage type."));
   }), jBinary;
@RReverser
Copy link
Member

It won't be portable across platforms. Why do you need it though? You can just not pass any callback if you prefer to, it will save file nevertheless.

@kpetrow
Copy link
Author

kpetrow commented May 5, 2017

I load in a point cloud(60-200MB binary final of int16's). Then query ~10 million position and apply the positions to the point cloud. The positions come in a stream from mysql server. Then at certain intervals write out the point cloud. I need to stop the mysql stream in order to output the file if being written async to ensure the point cloud isnt modified during output. When sync i can maintain the stream synchronously. I can do a pause in the stream, then pass in call back to resume when outputs the last file, but I think its really messed compared to sync write and doesnt save any time since I need to pause the database stream.

  CONNECTION_POOL = mysql.createConnection({
     connectionLimit: 10,
     host: server_name,
     user: user_name,
     password: password,
     database: schema,
     connectTimeout: 99999999999,
     multipleStatements: true,
     .........
     ..........
 });
 var myObjects, myObject = {};
 myObject.header = module.loadUsingJBin_and_other_stuff('up_to_200MbBinaryFile');
 myObject.myMassiveArray = module.loadUsingArrayBuffer_and_other_stuff('up_to_200MbBinaryFile');

 var positionStream = CONNECTION_POOL.query(myQuery)
 positionStream.on('result', function(currentPositionRow) {
     // this includes spawning multiple myObject's all with their own myMassiveArray
     myObjects = doStuff(myObject, currentPositionRow);

     // if time is write we need to output all the point clouds made
     if (currentPositionRow.time > outputTime && previousPositionRow.time < outputTime) {
         // We could add a puase here
         //CONNECTION_POOL.Pause();
         for (var i = 1; i < myObjects.length; i++) {
             // Conenction pool is paused, do i really want to deal with callback hell for async file write ?
             // callBack = (last) ? function (CONNECTION_POOL) { CONNECTION_POOL.Pause() } : 'NOT_TIME';
             module.WriteUsingJBin(outputTime, myObjects[i] /*, callBack*/ );
         }
     }
 })

I have tried cloning off the object to write but I get memory overflow, because the database stream and math is way faster then the write and garbage collection. Open for any suggestions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants