New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql,ui: support, test and optimize loads using thousands of databases #28376

Open
robert-s-lee opened this Issue Aug 8, 2018 · 5 comments

Comments

@robert-s-lee
Copy link

robert-s-lee commented Aug 8, 2018

FEATURE REQUEST

Motivation

A modern multi-tenant and micro services could have thousands of databases. Each database could represent a distinct customer and/or micro-services application. Within each database, a number of tables can be created.

Below is an extremely simple example of creating 10,000 databases with 1 table each and 3 rows per table for initial deployment.

Example

for i in `seq 10000`; do cockroach sql --insecure --database test -e "create database if not exists db${i}; use db${i}; create table if not exists test${i} (id int primary key); insert into test${i} values (1),(2),(3);"; done

Current limitations

Currently CockroachDB has many problems with a large number of databases:

  • the namespace tables and descriptors are propagated via gossip. with 10k databases and ten times that number of tables, this gossip traffic becomes huge

  • the lease management becomes inefficient at that scale

  • the databases section of the UI resource consumption is heavy

  • CPU consumption goes up on the server

screen shot 2018-08-08 at 8 58 17 am

@knz

This comment has been minimized.

Copy link
Member

knz commented Sep 17, 2018

The UI problem here is just the tip of the iceberg. Currently if you create 10k databases even the core product would have a large problem and not work properly.

I will requalify your issue to reveal its full scope.

@knz knz changed the title ui: handle thousands of databases more efficiently sql,ui: support, test and optimize loads using thousands of databases Sep 17, 2018

@knz

This comment has been minimized.

Copy link
Member

knz commented Sep 17, 2018

@vivekmenezes @awoods187 I think addressing this issue of scale will need intervention from multiple teams. It's hard for me to comprehend the scale upfront because there are so many moving pieces.

My advice would be to start with an investigation where we run a test cluster with that many databases (and corresponding table workload) and profile / inventory where resource consumption goes throughout the stack.

@vivekmenezes vivekmenezes added this to Triage in Schema changes via automation Sep 17, 2018

@knz

This comment has been minimized.

Copy link
Member

knz commented Sep 17, 2018

@vivekmenezes I disagree with you unassigning Piyush. There needs to be work in the web UI in addition to the core product. This will be separate from whatever schema management changes you're planning.

@piyush-singh

This comment has been minimized.

Copy link

piyush-singh commented Sep 17, 2018

Agreed! There are some tactical things that are specific to the UI that we can accomplish in 2.2 (and beyond) that will further this goal. Examples include tangentially related issues like better handling of large clusters' time series data (#30286) to more fundamental issues like the one Robert mentioned above around resource consumption and load times for large databases on the databases tab. This is a nice issue to be able to refer to when filing other issues that deal with scaling Cockroach

(thanks for reassigning me!)

@couchand

This comment has been minimized.

Copy link
Collaborator

couchand commented Sep 17, 2018

I suspect the UI part of this issue is a dupe of #22941 and will be covered by cleaning that up. But it's definitely useful putting that technical view of the issue in the context of real world usage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment