Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve roachprod --geo flag for easier geo-distributed testing #35866

Closed
awoods187 opened this issue Mar 18, 2019 · 1 comment · Fixed by #37492
Closed

Improve roachprod --geo flag for easier geo-distributed testing #35866

awoods187 opened this issue Mar 18, 2019 · 1 comment · Fixed by #37492
Assignees
Labels
A-benchmarking C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Comments

@awoods187
Copy link
Contributor

Roachprod --geo doesn't work with AWS:

roachprod create $CLUSTER -n 4 --clouds=aws  --aws-machine-type-ssd=c5d.4xlarge --geo
Error: accepts 1 arg(s), received 0

On GCP, roachprod will create a 4 node cluster the doubles up n1 and n2:

Andrews-MBP-2:~ andrewwoods$  roachprod create $CLUSTER -n 4 --gce-machine-type=n1-standard-16 --geo
Creating cluster andy-optzc with 4 nodes
OK
andy-optzc: [gce] 12h52m34s remaining
  andy-optzc-0001	andy-optzc-0001.us-east1-b.cockroach-ephemeral	10.142.0.159	35.229.126.6
  andy-optzc-0002	andy-optzc-0002.us-east1-b.cockroach-ephemeral	10.142.0.160	35.231.169.191
  andy-optzc-0003	andy-optzc-0003.us-west1-b.cockroach-ephemeral	10.138.0.31	35.230.95.249
  andy-optzc-0004	andy-optzc-0004.europe-west2-b.cockroach-ephemeral	10.154.0.31	35.234.156.121

This is problematic because we now need to point workload at n2 rather than n4 to get the geo-distributed cluster we desire. Note, if you point it at n1, then roachprod start won't work because it expects the cluster to be kicking off from n1.

Here is how you are forced to test with --geo:

roachprod create $CLUSTER -n 4 --gce-machine-type=n1-standard-16 --geo
roachprod run $CLUSTER --  "DEV=$(mount | grep /mnt/data1 | awk '{print $1}'); sudo umount /mnt/data1; sudo mount -o discard,defaults,nobarrier ${DEV} /mnt/data1/; mount | grep /mnt/data1"
roachprod stage $CLUSTER:1,3-4 cockroach
roachprod stage $CLUSTER:2 workload
roachprod start $CLUSTER:1,3-4
roachprod adminurl --open $CLUSTER:1
roachprod run $CLUSTER:1 -- "./cockroach workload fixtures import tpcc --warehouses=2500 --db=tpcc"
@awoods187 awoods187 added this to Triage in Core Performance via automation Mar 18, 2019
@awoods187 awoods187 added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-benchmarking labels Mar 18, 2019
@awoods187
Copy link
Contributor Author

awoods187 commented Mar 18, 2019

While we are at it, I'd like to be able to use the node map too which appears to require adding information to systems.locations
image
And the drill down:
image

@nvanbenschoten nvanbenschoten added this to Incoming in KV via automation Apr 16, 2019
@nvanbenschoten nvanbenschoten removed this from Triage in Core Performance Apr 16, 2019
craig bot pushed a commit that referenced this issue May 23, 2019
37492: roachprod: create geo-distributed clusters with extra nodes placed at the end r=nvanbenschoten a=ajwerner

roachprod: create geo-distributed clusters with extra nodes placed at the end                                                                                                                                      
                                                                                                                                                                                                                   
Prior to this change, roachprod would create geo-distributed clusters by                                                                                                                                           
placing nodes in AZs in contiguous chunks. If the number of total nodes                                                                                                                                            
was not evenly divisible by the number of regions, the first regions                                                                                                                                               
would be allocated one additional node. This allocation pattern is                                                                                                                                                 
rarely desirable. A user will commonly allocate a single extra node as a                                                                                                                                           
load generator and would generally like that load node to be the final                                                                                                                                             
node and for that final node to be the extra node.                                                                                                                                                                 
                                                                                                                                                                                                                   
This changes the allocation where the extra nodes are placed in the same                                                                                                                                           
regions as before but are given node indices at the end rather than with                                                                                                                                           
the other nodes in their region.                                                                                                                                                                                   
                                                                                                                                                                                                                   
After this change a cluster created with `roachprod create $CLUSTER -n 7 --geo`                                                                                                                                    
will look like:                                                                                                                                                                                                    
                                                                                                                                                                                                                   
```                                                                                                                                                                                                                
ajwerner-test-roachprod-gce: [gce] 12h47m58s remaining                                                                                                                                                             
  ajwerner-test-roachprod-gce-0001      ajwerner-test-roachprod-gce-0001.us-east1-b.cockroach-ephemeral 10.142.0.70      34.74.58.108                                                                              
  ajwerner-test-roachprod-gce-0002      ajwerner-test-roachprod-gce-0002.us-east1-b.cockroach-ephemeral 10.142.0.5       35.237.74.155                                                                             
  ajwerner-test-roachprod-gce-0003      ajwerner-test-roachprod-gce-0003.us-west1-b.cockroach-ephemeral 10.138.0.99      35.199.159.104                                                                            
  ajwerner-test-roachprod-gce-0004      ajwerner-test-roachprod-gce-0004.us-west1-b.cockroach-ephemeral 10.138.0.100     35.197.94.83                                                                              
  ajwerner-test-roachprod-gce-0005      ajwerner-test-roachprod-gce-0005.europe-west2-b.cockroach-ephemeral      10.154.15.237   35.230.143.190                                                                    
  ajwerner-test-roachprod-gce-0006      ajwerner-test-roachprod-gce-0006.europe-west2-b.cockroach-ephemeral      10.154.15.236   35.234.156.121                                                                    
  ajwerner-test-roachprod-gce-0007      ajwerner-test-roachprod-gce-0007.us-east1-b.cockroach-ephemeral 10.142.0.33      35.185.62.76                                                                              
```                                                                                                                                                                                                                
                                                                                                                                                                                                                   
Instead of the previous:                                                                                                                                                                                           
                                                                                                                                                                                                                   
```                                                                                                                                                                                                                
ajwerner-test-old: [gce] 12h19m21s remaining                                                                                                                                                                       
  ajwerner-test-old-0001        ajwerner-test-old-0001.us-east1-b.cockroach-ephemeral   10.142.0.139    34.74.150.216                                                                                              
  ajwerner-test-old-0002        ajwerner-test-old-0002.us-east1-b.cockroach-ephemeral   10.142.0.140    34.73.154.246                                                                                              
  ajwerner-test-old-0003        ajwerner-test-old-0003.us-east1-b.cockroach-ephemeral   10.142.0.141    35.243.176.131                                                                                             
  ajwerner-test-old-0004        ajwerner-test-old-0004.us-west1-b.cockroach-ephemeral   10.138.0.71     34.83.16.1                                                                                                 
  ajwerner-test-old-0005        ajwerner-test-old-0005.us-west1-b.cockroach-ephemeral   10.138.0.60     34.83.78.172                                                                                               
  ajwerner-test-old-0006        ajwerner-test-old-0006.europe-west2-b.cockroach-ephemeral       10.154.15.200    35.234.148.191                                                                                    
  ajwerner-test-old-0007        ajwerner-test-old-0007.europe-west2-b.cockroach-ephemeral       10.154.15.199    35.242.179.144                                                                                    
```                  
Fixes #35866.

Release note: None


37603: sql: add ALTER TABLE/INDEX .. UNSPLIT AT .. r=jeffrey-xiao a=jeffrey-xiao

Now that manual splits add a sticky bit to the range descriptor, and
the merge queue respects this sticky bit, we can expose functionality to
manually unset this sticky bit.

If the key to unsplit is not the start of a range, then the unsplit
command will throw an error. If the range was manually split (I.E. the
sticky bit is set), then the sticky bit will be unset. Otherwise, this
command is a no-op.

Syntactically, the unsplit command is identical to the split command.

Release note: None

Co-authored-by: Andrew Werner <ajwerner@cockroachlabs.com>
Co-authored-by: Jeffrey Xiao <jeffrey.xiao1998@gmail.com>
@craig craig bot closed this as completed in #37492 May 23, 2019
KV automation moved this from Incoming to Closed May 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-benchmarking C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants