Startup with many sites uses many db connections #336

Closed
mworrell opened this Issue May 23, 2012 · 9 comments

Comments

Projects
None yet
3 participants
@mworrell
Member

mworrell commented May 23, 2012

When starting zotonic we are using multiple db connections per site.

When there are many sites this may exhaust the available database connections.

We need to start sites slower, and less in parallel.

This also needs to be backported to zotonic 0.8.

@ghost ghost assigned mworrell May 23, 2012

@mworrell

This comment has been minimized.

Show comment
Hide comment
@mworrell

mworrell May 23, 2012

Member

I think we can use the jobs application for this.

Where we can define a maximum number of parallel jobs when starting.

Member

mworrell commented May 23, 2012

I think we can use the jobs application for this.

Where we can define a maximum number of parallel jobs when starting.

@mmzeeman

This comment has been minimized.

Show comment
Hide comment
@mmzeeman

mmzeeman May 24, 2012

Member

Hmm, maybe it is time to speed up the work on the new db layer. This uses poolboy, which you can configure it how many connections open at startup with size and max_overflow parameters.

Member

mmzeeman commented May 24, 2012

Hmm, maybe it is time to speed up the work on the new db layer. This uses poolboy, which you can configure it how many connections open at startup with size and max_overflow parameters.

@arjan

This comment has been minimized.

Show comment
Hide comment
@arjan

arjan Sep 12, 2012

Member

@mmzeeman how is it going with the new db layer?

Member

arjan commented Sep 12, 2012

@mmzeeman how is it going with the new db layer?

@mmzeeman

This comment has been minimized.

Show comment
Hide comment
@mmzeeman

mmzeeman Sep 12, 2012

Member

Didn"t touch it in a while. Busy. It sure would take a lot away a lot of installation hassles.

Member

mmzeeman commented Sep 12, 2012

Didn"t touch it in a while. Busy. It sure would take a lot away a lot of installation hassles.

@arjan

This comment has been minimized.

Show comment
Hide comment
@arjan

arjan Sep 12, 2012

Member

Do you have a branch somewhere? I might have time the coming weeks to take a look at this.
Really would like multi-db support... :)

Member

arjan commented Sep 12, 2012

Do you have a branch somewhere? I might have time the coming weeks to take a look at this.
Really would like multi-db support... :)

@mmzeeman

This comment has been minimized.

Show comment
Hide comment
@mmzeeman

mmzeeman Sep 12, 2012

Member

No zotonic branch yet. Was just busy with building the layer. Last thing I did was experiment with transactions and sending results asynchronous to a pid. That makes doing transactions a bit hard though.

Sent from my iPhone

On 12 sep. 2012, at 20:48, Arjan Scherpenisse notifications@github.com wrote:

Do you have a branch somewhere? I might have time the coming weeks to take a look at this.
Really would like multi-db support... :)


Reply to this email directly or view it on GitHub.

Member

mmzeeman commented Sep 12, 2012

No zotonic branch yet. Was just busy with building the layer. Last thing I did was experiment with transactions and sending results asynchronous to a pid. That makes doing transactions a bit hard though.

Sent from my iPhone

On 12 sep. 2012, at 20:48, Arjan Scherpenisse notifications@github.com wrote:

Do you have a branch somewhere? I might have time the coming weeks to take a look at this.
Really would like multi-db support... :)


Reply to this email directly or view it on GitHub.

@mworrell

This comment has been minimized.

Show comment
Hide comment
@mworrell

mworrell Sep 12, 2012

Member

I can try to give a shot at this issue.

It is a kind of generic problem where we need to be able to recycle connections between sites when we are "starving".

Member

mworrell commented Sep 12, 2012

I can try to give a shot at this issue.

It is a kind of generic problem where we need to be able to recycle connections between sites when we are "starving".

@mworrell

This comment has been minimized.

Show comment
Hide comment
@mworrell

mworrell Sep 12, 2012

Member

Do we have a trace/error-log entry of this issue?

Member

mworrell commented Sep 12, 2012

Do we have a trace/error-log entry of this issue?

@mworrell

This comment has been minimized.

Show comment
Hide comment
@mworrell

mworrell Apr 22, 2015

Member

This should be fixed by refactoring the OTP supervision tree and the site-startup process.

Closing for now.

Member

mworrell commented Apr 22, 2015

This should be fixed by refactoring the OTP supervision tree and the site-startup process.

Closing for now.

@mworrell mworrell closed this Apr 22, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment