Unmatched error code on Linux 6.2 for "connect()"

[jessehui]

Considering this case:

A client socket tries to connect to a non-listening port of a server socket. Based on my knowledge, the client will receive an RST and the errno is set to ECONNREFUSED (which is the behavior implemented in Linux). On the second try, the behavior should be the same and errno is still ECONNREFUSED.

This behavior is the same for sync "connect()" on both the 5.15 and 6.2 kernels and the io_uring connect on Linux 5.15. But for the 6.2 kernel, the second io_uring connect returns ECONNABORTED.

I made a minimal test to demonstrate this and will be put at the end of this issue. I would like to know if I am doing anything wrong or if this is a bug or a known issue or if it is intended. Thank you!

Code to reproduce:

#include <stdio.h>
#include <netinet/in.h>
#include <string.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>
#include <liburing.h>
#include <arpa/inet.h>

#define THROW_ERROR(fmt, ...)   do { \
    printf("\t\tERROR:" fmt " in func %s at line %d of file %s with errno %d: %s\n", \
    ##__VA_ARGS__, __func__, __LINE__, __FILE__, errno, strerror(errno)); \
    return -1; \
} while (0)

static int ENTRIES = 128;
static struct io_uring ring;
static int local_errno = 0;

static void sigint_handler();
static int get_completion_and_print(struct io_uring *ring);

static int init_test_server(struct sockaddr_in *serv_addr) {
    // Init server socker. Bind but don't listen.
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd < 0) {
        THROW_ERROR("server_routine, error creating socket");
    }

    serv_addr->sin_family = AF_INET;
    serv_addr->sin_addr.s_addr = inet_addr("127.0.0.1");

    socklen_t servaddr_len = sizeof(struct sockaddr_in);
    if (bind(sockfd, (struct sockaddr *)serv_addr, servaddr_len) < 0) {
        THROW_ERROR("server_routine, error binding socket");
    }

    // Get the addresses the socket is bound to because the port is chosen by the network stack.
    if (getsockname(sockfd, (struct sockaddr *)serv_addr, &servaddr_len) < 0) {
        THROW_ERROR("getsockname() failed");
    }

    return 0;
}

// Return the client fd
static int init_test_client() {
    // Init client socket
    struct sockaddr_in client_addr = {};
    int clientfd = socket(AF_INET, SOCK_STREAM, 0);
    if (clientfd < 0) {
        THROW_ERROR("server_routine, error creating socket");
    }

    client_addr.sin_family = AF_INET;
    client_addr.sin_addr.s_addr = inet_addr("127.0.0.1");

    socklen_t addr_len = sizeof(struct sockaddr_in);
    if (bind(clientfd, (struct sockaddr *)&client_addr, addr_len) < 0) {
        THROW_ERROR("server_routine, error binding socket");
    }

    // Get the addresses the socket is bound to because the port is chosen by the network stack.
    if (getsockname(clientfd, (struct sockaddr *)&client_addr, &addr_len) < 0) {
        THROW_ERROR("getsockname() failed");
    }

    return clientfd;
}

static int test_io_uring_connect(int clientfd, struct sockaddr_in *serv_addr) {
    struct io_uring_sqe *sqe;
    sqe = io_uring_get_sqe(&ring);

    io_uring_prep_connect(sqe, clientfd, (const struct sockaddr *)serv_addr,
                          sizeof(struct sockaddr_in));

    int sub_num = io_uring_submit_and_wait(&ring, 1);
    printf("submit number = %d\n", sub_num);

    int ret = get_completion_and_print(&ring);
    if (ret != 1 || local_errno != ECONNREFUSED) {
        THROW_ERROR("get completion and print error");
    }

    return 0;
}

int main() {
    struct io_uring_params params;
    int client_socket;

    if (geteuid()) {
        THROW_ERROR("You need root privileges to run this program.\n");
    }

    memset(&params, 0, sizeof(params));
    params.flags |= IORING_SETUP_SQPOLL;
    params.sq_thread_idle = 120000; // 2 minutes in ms

    int ret = io_uring_queue_init_params(ENTRIES, &ring, &params);
    if (ret < 0) {
        THROW_ERROR("iouring init failure");
    }

    signal(SIGINT, sigint_handler);

    struct sockaddr_in serv_addr = {};
    ret = init_test_server(&serv_addr);
    if (ret < 0) {
        THROW_ERROR("init test server failure");
    }

    int clientfd = init_test_client();
    if (clientfd < 0) {
        THROW_ERROR("prepare client failure");
    }

    // Try connect twice. Expect to get the ECONNREFUSED errno same as sync "connect()".
    ret = test_io_uring_connect(clientfd, &serv_addr);
    if (ret < 0) {
        THROW_ERROR("test connect failure");
    }

    printf("second try\n");

    ret = test_io_uring_connect(clientfd, &serv_addr);
    if (ret < 0) {
        THROW_ERROR("test connect failure");
    }

    printf("TEST PASSED!\n");
    return 0;
}

void sigint_handler() {
    printf("^C pressed. Shutting down.\n");
    io_uring_queue_exit(&ring);
    exit(0);
}


int get_completion_and_print(struct io_uring *ring) {
    struct io_uring_cqe *cqe;
    int ret = io_uring_wait_cqe(ring, &cqe);
    /* Mark this completion as seen */
    io_uring_cqe_seen(ring, cqe);

    if (ret < 0) {
        perror("io_uring_wait_cqe");
        return -1;
    }

    if (cqe->res < 0) {
        /* The system call invoked asynchonously failed */
        printf("io_uring_wait_cqe: %s\n", strerror(-cqe->res));
        local_errno = -cqe->res;
        return 1;
    }

    printf("connect request complete, ret = %d\n", cqe->res);
    return 0;
}

On Linux 5.15:

On Linux 6.2:

[axboe]

I suspect the same will happen if you change it to sync connect AND set O_NONBLOCK on the clientfd after opening the socket. Can you test?

BTW, SQPOLL does not require root in any io_uring kernel you'd want to use. This includes 5.10-stable, 5.15-stable, and anything after those.

[axboe]

OK this does look a bit odd... If you're able to compile and test your own kernels, please try:

diff --git a/io_uring/net.c b/io_uring/net.c
index b7f190ca528e..b2e7ffdabd11 100644
--- a/io_uring/net.c
+++ b/io_uring/net.c
@@ -1461,7 +1461,8 @@ int io_connect(struct io_kiocb *req, unsigned int issue_flags)
 
 	ret = __sys_connect_file(req->file, &io->address,
 					connect->addr_len, file_flags);
-	if ((ret == -EAGAIN || ret == -EINPROGRESS) && force_nonblock) {
+	if ((ret == -EAGAIN || ret == -EINPROGRESS || ret == -ECONNABORTED)
+	    && force_nonblock) {
 		if (ret == -EINPROGRESS) {
 			connect->in_progress = true;
 		} else {

which does pass your test case. I'll clean that up and add it too.

[axboe]

Here's where I ended up:

https://git.kernel.dk/cgit/linux-block/commit/?h=io_uring-6.3&id=f0385a0c84624a35fd88901112d6bae275f0ca44

I'll send it out for review. If you want a Reviewed-by: attribution, I'll need your name and email to add to it. Completely up to you. I already referenced this bug in the commit.

[axboe]

And here's the liburing test case:

https://git.kernel.dk/cgit/liburing/commit/?id=7389eb1e4192132289e4ee9ccecb5bd00db888e0

[jessehui]

Thank you for your quick response! It's very kind of you to mention me with that attribution.
Here is my info: Hui, Chunyang <sanqian.hcy@antgroup.com> .
I will do the test today and will give an update when I get the result. Thank you!

[axboe]

Sounds good, thanks! I've added the reported-by to the commit and pushed out a new one.

[jessehui]

Thank you for your help! Additionally, I was wondering if kernel versions 6.0 or 6.1 are expected to pass the test. Would you happen to know when this might be introduced? Our team is in need of a kernel version no lower than 6.0 for deployment, so any insight you have would be greatly appreciated. If not, do you think we can anticipate using version 6.3 instead? Thank you again.

[jessehui]

Thank you so much for your prompt work. I've tested the kernel 6.2.1 with the patch you provided, and it works perfectly. Please feel free to close this issue.

[axboe]

Perfect, thanks for testing! And for the report in the first place, with the reproducer.

In terms of kernel version, this will go into the 6.3-rc series this week, but will then makes its way into the stable branches as well. So if you based on 6.2-stable, you should be fine.