Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CPU Usage to 100% #197

Open
motleycrew opened this issue Jun 26, 2017 · 5 comments
Open

CPU Usage to 100% #197

motleycrew opened this issue Jun 26, 2017 · 5 comments

Comments

@motleycrew
Copy link

motleycrew commented Jun 26, 2017

I am using phantom-pool which interns uses generic-pool.

After running few hundered urls, phantomJs CPU usage shoots up to 100% and instances are not killed. Is anybody else seeing this? I have to manually restart the pool. Is there a better way to fix this?

Is there a way to kill old processes let say after 1 minute which have stopped responding?

My current configuration bellow:

self.pool = phantomPool.default({
max: 5, // default
min: 1, // default
// how long a resource can stay idle in pool before being removed
idleTimeoutMillis: 30000, // default.
// maximum number of times an individual resource can be reused before being destroyed; set to 0 to disable
maxUses: 10, // default
// function to validate an instance prior to use; see https://github.com/coopernurse/node-pool#createpool
validator: () => Promise.resolve(true), // defaults to always resolving true
// validate resource before borrowing; required for maxUses andvalidatortestOnBorrow: true, // default // For all opts, see opts at https://github.com/coopernurse/node-pool#createpool phantomArgs: [phantomArgs, { logLevel: 'error', }], // arguments passed to phantomjs-node directly, default is[]`. For all opts, see https://github.com/amir20/phantomjs-node#phantom-object-api
@t3hmrman
Copy link
Contributor

t3hmrman commented Mar 29, 2019

I think I might have found something that might explain this -- you need to make sure your create promise does not fail.

I just spent some time debugging this issue, and it looks like node-pool will retry create in a tight loop, so if your pool does something expensive (which is why you're pooling in the first place), it will cause memory to run away.

You need to set a handler for factoryCreateError -- if you don't, the create just gets retried endlessly. I added some code to stop the pool:

        this.pool.on("factoryCreateError", err => {
            this.poolErrors.push(err);
            // Stop the pool if resource allocation fails too many times
            if (this.poolErrors.length > this.maxPoolErrors) {
                this.emit("error", new Errors.ResourceAllocationFailureLimitExceeded());
                this.disconnect(); // stops the pool and does some other stuff
            }
        });

@alolis
Copy link

alolis commented May 28, 2019

It seems that others (including myself) have this issue if you search for "infinite" in Issues

@ImHype
Copy link

ImHype commented Jun 3, 2019

I think it is a good default behavior which can solve this problem.

this._pool.on('factoryCreateError', err => {
       this._pool._waitingClientsQueue.dequeue().reject(err);
});

@mattfysh
Copy link

this._pool.on('factoryCreateError', err => {
       this._pool._waitingClientsQueue.dequeue().reject(err);
});

This doesn't seem safe as there's no guarantee the error came from the first waiting client in the queue.

@RogerKang
Copy link

this._pool.on('factoryCreateError', err => {
       this._pool._waitingClientsQueue.dequeue().reject(err);
});

This doesn't seem safe as there's no guarantee the error came from the first waiting client in the queue.

In my understanding, _waitingClientsQueue should be a list waiting for new clients which being creating, when an error is thrown in creating, give this error as rejection in any item in this _waitingClientsQueue should be ok to notice that this waiting will not succeed due to this create error. And naturally, pick the first one from the queue to reject will be the common way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants