Scaling Up of GlusterFS-strorage only add new peer without new bricks in Openshift

[imranrazakhan]

Kind of issue

Bug/Feature request

Observed behavior

I started with one node Openshift cluster and it successfully deployed master/node and gluster volume. Now i extend Openshift cluster and it was successfully.

but on extending glusterfs volume with below

inventory file look like below

    openshift_storage_glusterfs_is_native=false
    openshift_storage_glusterfs_storageclass=true
    openshift_storage_glusterfs_heketi_is_native=true
    openshift_storage_glusterfs_heketi_executor=ssh
    openshift_storage_glusterfs_heketi_ssh_port=22
    openshift_storage_glusterfs_heketi_ssh_user=root
    openshift_storage_glusterfs_heketi_ssh_sudo=false
    openshift_storage_glusterfs_heketi_ssh_keyfile="/root/.ssh/id_rsa"

    [glusterfs]
    10.1.1.1  glusterfs_devices='[ "/dev/vdb" ]'
    10.1.1.2  glusterfs_devices='[ "/dev/vdb" ]'  openshift_node_labels="type=upgrade"

ansible-playbook -i inventory2.ini /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/config.yml -e openshift_upgrade_nodes_label="type=upgrade"

it only added 10.1.1.2 as peer but volume still has only one brick

Following customization done to start deploy gluster from 1 node {--durability none}

openshift-ansible/roles/openshift_storage_glusterfs/tasks/heketi_init_db.yml

- name: Create heketi DB volume
  command: "{{ glusterfs_heketi_client }} setup-openshift-heketi-storage --image {{ glusterfs_heketi_image }} --listfile /tmp/heketi-storage.json **--durability none**"
  register: setup_storage

gluster peer status
Number of Peers: 1
Hostname: 10.1.1.2
Uuid: 1b8159e4-99e2-4f4d-ad95-e97bc8655d32
State: Peer in Cluster (Connected)

gluster volume info

Volume Name: heketidbstorage
Type: Distribute
Volume ID: 769419b9-d28f-4cdd-a8f3-708b6b738f65
Status: Started
Snapshot Count: 0
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.1.1.1:/var/lib/heketi/mounts/vg_4187bfa3eb090ceffea9c53b156ddbd4/brick_80401b43be8c3c8a74417b18ad574524/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

Expected/desired behavior

I am expecting that on addition of every new node it should create new brick too

Details on how to reproduce (minimal and precise)

Add nodes in gluster cluster with below commands
ansible-playbook -i inventory2.ini /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/config.yml -e openshift_upgrade_nodes_label="type=upgrade"

Information about the environment:

• Heketi version used (e.g. v6.0.0 or master): OpenShift 3.10
• Operating system used: CentOS
• Heketi compiled from sources, as a package (rpm/deb), or container: Container
• If container, which container image: docker.io/heketi/heketi:latest
• Using kubernetes, openshift, or direct install: Openshift
• If kubernetes/openshift, is gluster running inside kubernetes/openshift or outside: outside
• If kubernetes/openshift, how was it deployed (gk-deploy, openshift-ansible, other, custom): openshift-ansible

Other useful information

[phlogistonjohn]

I am expecting that on addition of every new node it should create new brick too

Unfortunately, this is not how the system works today. If a volume is created with "distribute" and no replicas the volume will always be that way. Adding new nodes will allow you to create replica-3 volumes but the types of existing volumes are fixed.

[imranrazakhan]

1- any suggestion to customize current openshift ansible script to achieve it?

Adding new nodes will allow you to create replica-3 volumes
2- It mean if we start with 3 nodes and volume-type is replicate then on adding fourth node it will create brick otherwise not? correct me if i am wrong.

[phlogistonjohn]

1- any suggestion to customize current openshift ansible script to achieve it?

No, this is not really feasible with the the architecture of the system today.

Adding new nodes will allow you to create replica-3 volumes

2- It mean if we start with 3 nodes and volume-type is replicate then on adding fourth node it will create brick otherwise not? correct me if i am wrong.

No, adding nodes will not change the replica count of existing volumes. And certainly not automatically.

If you deploy a 3 node cluster from the beginning and create heketidbstorage on it it will always be a replica-3 volume. You can then then add bricks to that volume via the heketi-cli volume expand command but this will only add new sets of bricks (distribute), not an additional brick to the replica set. Heketi simply doesn't have what you're asking for today.

I strongly suggest simply starting out with a 3-node cluster so you can have a replica-3 volume from the beginning. If you absolutely have to start with a one node cluster and then add nodes you're going to have to do a lot of manual & tricky steps that are not anything we typically support.

[imranrazakhan]

You can then then add bricks to that volume via the heketi-cli volume expand command but this will only add new sets of bricks (distribute), not an additional brick to the replica set. Heketi simply doesn't have what you're asking for today.
Can we add brick to replica set manually with gluster commands, what would be impact on heketi?

[phlogistonjohn]

I checked with one of the gluster devs and the answer is a qualified yes. The issue you'll have is that heketi will not understand the structural changes you'd be making. You'd either have to live with a heketi that is always out of sync with gluster (and the issues this will cause in createing new volumes,etc) OR you'd have to manually edit the heketi db to add in the bricks and carefully change all the data structures and sizes in the db. This would not be the kind of thing we'd be able to help much with supporting.

[imranrazakhan]

@phlogistonjohn already one issue created for this Feature, Hope so will get it soon

#963

[phlogistonjohn]

Hi @imranrazakhan since the other issue you mention seems to encapsulate your request I'm going to close this issue (duplicate). If I misunderstood please feel free to reopen this issue as needed.