Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

glusterd: volume start doesn't fail with properly if the port range is exhausted #2910

Closed
nik-redhat opened this issue Oct 28, 2021 · 0 comments · Fixed by #2920
Closed

glusterd: volume start doesn't fail with properly if the port range is exhausted #2910

nik-redhat opened this issue Oct 28, 2021 · 0 comments · Fixed by #2920
Assignees
Labels

Comments

@nik-redhat
Copy link
Contributor

Description of problem:
gluster v start <volname> doesn't fail correctly if the port range is exhausted (or after limiting the value of max-port in the glusterd.vol file)
It should have failed with a correct error message i.e, "All the ports in the range are exhausted" in the log file.
glusterd hangs in the above scenario and needs to be restarted with the volfile updated to allow all the bricks to be started.

RCA
After the latest update to use random ports in pmap, we are executing the below code to check for free ports:

int
pmap_port_alloc(xlator_t *this)
{
    struct pmap_registry *pmap = NULL;
    int p = 0;

    GF_ASSERT(this);

    pmap = pmap_registry_get(this);

    while (true) {
        /* coverity[DC.WEAK_CRYPTO] */
        p = (rand() % (pmap->max_port - pmap->base_port + 1)) + pmap->base_port;
        if (pmap_port_isfree(p)) {
            break;
        }
    }

    return p;
}

The issue in the above code is if the port range is exhausted it would never break/return and it will become an infinite loop.
We need to either limit the number of times the check is done or use a list/array to exit the loop if all the ports in the base_port to max_port range are checked.

The issue was noticed in the latest gluster-10 functional test on redant framework for the test case - test_reserved_port_range_for_gluster..

@nik-redhat nik-redhat self-assigned this Oct 28, 2021
nik-redhat added a commit to nik-redhat/glusterfs that referenced this issue Nov 2, 2021
**Issue:**
After the latest update to use randomizeed ports, if all ports
till the max-port value in the glusterd.vol are occupied then,
volume start doesn't fails with the error that "All ports are
exhausted", rather it hangs. The reason being there ins't any check
in the function `pmap_port_alloc` to check for the occupied ports
within the permissible range and the loop goes on infinetely.

**Fix:**:
Added a check for tha available ports within the max-port range
and fail with correct error msg (already implemented) if all ports
are occupied.

Fixes: gluster#2910

Change-Id: I862822ce573c8b78cf0fb5dd2db95667aa4581d3
Signed-off-by: nik-redhat <nladha@redhat.com>
nik-redhat added a commit to nik-redhat/glusterfs that referenced this issue Nov 2, 2021
**Issue:**
After the latest update to use randomized ports, if all ports
till the max-port value in the glusterd.vol are occupied then,
volume start doesn't fails with the error that "All ports are
exhausted", rather it hangs. The reason is there ins't any check
in the function `pmap_port_alloc` to check for the occupied ports
within the permissible range and the loop goes on infinitely.

**Fix:**:
Added a check for the available ports within the max-port range
and fail with correct error msg (already implemented) if all ports
are occupied.

Fixes: gluster#2910

Change-Id: I862822ce573c8b78cf0fb5dd2db95667aa4581d3
Signed-off-by: nik-redhat <nladha@redhat.com>
nik-redhat added a commit to nik-redhat/glusterfs that referenced this issue Nov 2, 2021
**Issue:**
After the latest update to use randomized ports, if all ports
till the max-port value in the glusterd.vol are occupied then,
volume start doesn't fails with the error that "All ports are
exhausted", rather it hangs. The reason is there ins't any check
in the function `pmap_port_alloc` to check for the occupied ports
within the permissible range and the loop goes on infinitely.

**Fix:**:
Added a check for the available ports within the max-port range
and fail with correct error msg (already implemented) if all ports
are occupied.

Fixes: gluster#2910

Change-Id: I862822ce573c8b78cf0fb5dd2db95667aa4581d3
Signed-off-by: nik-redhat <nladha@redhat.com>
nik-redhat added a commit to nik-redhat/glusterfs that referenced this issue Nov 19, 2021
**Issue:**
After the latest update to use randomized ports, if all ports
till the max-port value in the glusterd.vol are occupied then,
volume start doesn't fails with the error that "All ports are
exhausted", rather it hangs. The reason is there ins't any check
in the function `pmap_port_alloc` to check for the occupied ports
within the permissible range and the loop goes on infinitely.

**Fix:**:
Added a check for the available ports within the max-port range
and fail with correct error msg (already implemented) if all ports
are occupied.

Fixes: gluster#2910

Change-Id: I862822ce573c8b78cf0fb5dd2db95667aa4581d3
Signed-off-by: nik-redhat <nladha@redhat.com>
nik-redhat added a commit to nik-redhat/glusterfs that referenced this issue Nov 21, 2021
**Issue:**
After the latest update to use randomized ports, if all ports
till the max-port value in the glusterd.vol are occupied then,
volume start doesn't fails with the error that "All ports are
exhausted", rather it hangs. The reason is there ins't any check
in the function `pmap_port_alloc` to check for the occupied ports
within the permissible range and the loop goes on infinitely.

**Fix:**:
Added a check for the available ports within the max-port range
and fail with correct error msg (already implemented) if all ports
are occupied.

Fixes: gluster#2910

Change-Id: I862822ce573c8b78cf0fb5dd2db95667aa4581d3
Signed-off-by: nik-redhat <nladha@redhat.com>
xhernandez pushed a commit that referenced this issue Nov 22, 2021
**Issue:**
After the latest update to use randomized ports, if all ports
till the max-port value in the glusterd.vol are occupied then,
volume start doesn't fails with the error that "All ports are
exhausted", rather it hangs. The reason is there ins't any check
in the function `pmap_port_alloc` to check for the occupied ports
within the permissible range and the loop goes on infinitely.

**Fix:**:
Added a check for the available ports within the max-port range
and fail with correct error msg (already implemented) if all ports
are occupied.

Fixes: #2910

Change-Id: I862822ce573c8b78cf0fb5dd2db95667aa4581d3
Signed-off-by: nik-redhat <nladha@redhat.com>
nik-redhat added a commit to nik-redhat/glusterfs that referenced this issue Dec 6, 2021
**Issue:**
After the latest update to use randomized ports, if all ports
till the max-port value in the glusterd.vol are occupied then,
volume start doesn't fails with the error that "All ports are
exhausted", rather it hangs. The reason is there ins't any check
in the function `pmap_port_alloc` to check for the occupied ports
within the permissible range and the loop goes on infinitely.

**Fix:**:
Added a check for the available ports within the max-port range
and fail with correct error msg (already implemented) if all ports
are occupied.

Updates: gluster#2910

>Fixes: gluster#2910

>Change-Id: I862822ce573c8b78cf0fb5dd2db95667aa4581d3
>Signed-off-by: nik-redhat <nladha@redhat.com>

Change-Id: I3b5a6b985a5d5010be238585edde687b745dc8c5
Signed-off-by: nik-redhat <nladha@redhat.com>
Shwetha-Acharya pushed a commit that referenced this issue Jan 7, 2022
**Issue:**
After the latest update to use randomized ports, if all ports
till the max-port value in the glusterd.vol are occupied then,
volume start doesn't fails with the error that "All ports are
exhausted", rather it hangs. The reason is there ins't any check
in the function `pmap_port_alloc` to check for the occupied ports
within the permissible range and the loop goes on infinitely.

**Fix:**:
Added a check for the available ports within the max-port range
and fail with correct error msg (already implemented) if all ports
are occupied.

Updates: #2910

>Fixes: #2910

>Change-Id: I862822ce573c8b78cf0fb5dd2db95667aa4581d3
>Signed-off-by: nik-redhat <nladha@redhat.com>

Change-Id: I3b5a6b985a5d5010be238585edde687b745dc8c5
Signed-off-by: nik-redhat <nladha@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant