Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UCT/SOCKCM: sockcm implementation over skeleton #3570

Open
wants to merge 114 commits into
base: master
from

Conversation

@Akshay-Venkatesh
Copy link
Contributor

commented May 8, 2019

What

Builds sockets based connection functionality over sockets connection management skeleton.

Notes

Some of the functionality is pretty much a copy-paste from rdmacm implementation.

@dmitrygx Can you recommend where I should be using the new non-blocking socket operations you've introduced recently?

cc @yosefe @brminich @evgeny-leksikov

@swx-jenkins

This comment has been minimized.

Copy link
Collaborator

commented May 8, 2019

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented May 8, 2019

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/10147/ for details (Mellanox internal link).

@bureddy

This comment has been minimized.

Copy link
Contributor

commented May 9, 2019

12:03:21 [ RUN      ] udx/test_ucp_sockaddr_with_rma_atomic.wireup/3
12:03:21 [     INFO ] Testing 1.1.1.5:35247
12:07:44 [hpc-test-node:7800 :1:12850] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
12:07:44 
12:07:44 /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-test-node/worker/1/contrib/../src/uct/ib/rdmacm/rdmacm_iface.c: [ uct_rdmacm_iface_process_event() ]
12:07:44       ...
12:07:44       379     /* client error events */
12:07:44       380     case RDMA_CM_EVENT_UNREACHABLE:
12:07:44       381         hdr = *(uct_rdmacm_priv_data_hdr_t *)event->param.conn.private_data;
12:07:44 ==>   382         if ((event->param.conn.private_data_len > 0) &&
12:07:44       383             (hdr.status == UCS_ERR_REJECTED)) {
12:07:44       384             ucs_assert(hdr.length == 0);
12:07:44       385             ucs_assert(event->param.conn.private_data_len >= sizeof(hdr));
12:07:44 
12:07:44 ==== backtrace (tid:  12850) ====
12:07:44  0 0x0000000000003c85 uct_rdmacm_iface_process_event()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-test-node/worker/1/contrib/../src/uct/ib/rdmacm/rdmacm_iface.c:382
12:07:44  1 0x000000000000447b uct_rdmacm_iface_event_handler()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-test-node/worker/1/contrib/../src/uct/ib/rdmacm/rdmacm_iface.c:438
12:07:44  2 0x000000000003f08b ucs_async_handler_dispatch()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-test-node/worker/1/contrib/../src/ucs/async/async.c:223
12:07:44  3 0x000000000003f388 ucs_async_dispatch_handlers()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-test-node/worker/1/contrib/../src/ucs/async/async.c:252
12:07:44  4 0x00000000000428bd ucs_async_thread_func()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-test-node/worker/1/contrib/../src/ucs/async/thread.c:113
12:07:44  5 0x0000000000007e25 start_thread()  pthread_create.c:0
12:07:44  6 0x00000000000f834d __clone()  ???:0
12:07:44 =================================
12:07:44 Sending notification to akvenkatesh@nvidia.com
12:07:45 [hpc-test-node:7800 :1:12850] Process frozen...
14:27:25 
@Akshay-Venkatesh

This comment has been minimized.

Copy link
Contributor Author

commented May 9, 2019

@bureddy I'm not sure why we see this error in rdmacm transport since we're not touching that path.

@yosefe

This comment has been minimized.

Copy link
Contributor

commented May 9, 2019

@evgeny-leksikov @alinask pls take a look on the error, maybe sockcm exposes some problem with UCP client/server having more than one possible UCT which supports it

@Akshay-Venkatesh Akshay-Venkatesh marked this pull request as ready for review May 10, 2019

@yosefe yosefe requested review from alinask and evgeny-leksikov May 12, 2019

src/uct/tcp/sockcm/sockcm_ep.c Outdated Show resolved Hide resolved
src/uct/tcp/sockcm/sockcm_def.h Outdated Show resolved Hide resolved
src/uct/tcp/sockcm/sockcm_ep.c Outdated Show resolved Hide resolved
}

static ucs_status_t uct_sockcm_iface_reject(uct_iface_h tl_iface,
uct_conn_request_h conn_request)
{
return UCS_ERR_NOT_IMPLEMENTED;
return UCS_OK;

This comment has been minimized.

Copy link
@alinask

alinask May 13, 2019

Contributor

there is no indication to the client of a reject?
the client needs to know of a reject from the server and invoke the error handling flow

This comment has been minimized.

Copy link
@Akshay-Venkatesh

Akshay-Venkatesh Jun 4, 2019

Author Contributor

Client gets to know that connection attempt didn't succeed from this https://github.com/openucx/ucx/pull/3570/files#diff-aca878e51b7d5e6eaa75cd20a1375eadR157

Do I need to explicitly handle reject besides this ^ ?

This comment has been minimized.

Copy link
@alinask

alinask Jun 5, 2019

Contributor

Yes, the client needs to invoke the error handling flow to handle the rejection from the server. Some of the gtests test this.
@evgeny-leksikov

hdr = &conn_param.hdr;

/* pack worker address into private data */
hdr->length = ep->pack_cb(ep->pack_cb_arg, "eth0",

This comment has been minimized.

Copy link
@alinask

alinask May 13, 2019

Contributor

why is eth0 hardcoded? can't it be another interface?

This comment has been minimized.

Copy link
@Akshay-Venkatesh

Akshay-Venkatesh Jun 4, 2019

Author Contributor

Device name isn't relevant for sockcm when listening address is set to INADDR_ANY. This may not be valid when a specific address is set. This will need some verification. Is there a test in gtest to check if things work as expected?

I've replaced with DEV_NAME_NULL for now.

This comment has been minimized.

Copy link
@alinask

alinask Jun 5, 2019

Contributor

In gtest, we go over the available interfaces and for the sockaddr resources (interfaces) we take the ones that are IPoIB (and active and have a valid sa_family)
https://github.com/openucx/ucx/blob/master/test/gtest/ucp/test_ucp_sockaddr.cc#L103
https://github.com/openucx/ucx/blob/master/test/gtest/uct/uct_test.cc#L233
So now that you are adding a new sockaddr transport, non IPoIB interfaces should be tested as well.

src/uct/tcp/sockcm/sockcm_iface.c Outdated Show resolved Hide resolved
uct_sockcm_conn_param_t conn_param)
{
iface->conn_request_cb(&iface->super.super, iface->conn_request_arg, NULL,
conn_param.private_data, conn_param.private_data_len);

This comment has been minimized.

Copy link
@alinask

alinask May 13, 2019

Contributor

conn_param.private_data points to the data itself without the hdr, so how would the upper layer know how much was actually wirtten? hdr->length holds this value.

This comment has been minimized.

Copy link
@Akshay-Venkatesh

Akshay-Venkatesh Jun 4, 2019

Author Contributor

Missing full context here. Will get back on this

This comment has been minimized.

Copy link
@alinask

alinask Jul 28, 2019

Contributor

did you get a chance to come back to this?

This comment has been minimized.

Copy link
@Akshay-Venkatesh

Akshay-Venkatesh Jul 29, 2019

Author Contributor

RDMACM uses the following:

static void uct_rdmacm_iface_process_conn_req(uct_rdmacm_iface_t *iface,
                                              struct rdma_cm_event *event,
                                              struct sockaddr *remote_addr)
{
    uct_rdmacm_priv_data_hdr_t *hdr;

    hdr = (uct_rdmacm_priv_data_hdr_t*) event->param.ud.private_data;
    ucs_assert(hdr->status == UCS_OK);

    /* TODO check the iface's cb_flags to determine when to invoke this callback.
     * currently only UCT_CB_FLAG_ASYNC is supported so the cb is invoked from here */
    iface->conn_request_cb(&iface->super.super, iface->conn_request_arg,
                           /* connection request*/
                           event,
                           /* private data */
                           UCS_PTR_BYTE_OFFSET(event->param.ud.private_data,
                                               sizeof(uct_rdmacm_priv_data_hdr_t)),
                           /* length */
                           hdr->length);
}

SOCKCM uses the following instead:

static void uct_sockcm_iface_process_conn_req(uct_sockcm_iface_t *iface,
                                             uct_sockcm_conn_param_t conn_param)
{
    int *sock_fd;
    sock_fd = ucs_malloc(sizeof(int), "sock_fd");

    *sock_fd = conn_param.fd;
    ucs_debug("process conn req: accepted fd %d %m", *sock_fd);

    iface->conn_request_cb(&iface->super.super, iface->conn_request_arg, sock_fd,
               conn_param.private_data, conn_param.private_data_len);
}

But I also set conn_param.private_data_len to have the same value as hdr->length here

Will that work?

This comment has been minimized.

Copy link
@alinask

alinask Jul 30, 2019

Contributor

But conn_param.private_data_len should include all the sent data length, right? including the hdr itself plus the actual data. So that the transport would pass both hdr and data to the remote side.
conn_param.private_data_len = sizeof(*hdr) + hdr->length
but for the callback, the upper layer only cares about the actual data: hdr->length

This comment has been minimized.

Copy link
@Akshay-Venkatesh

Akshay-Venkatesh Jul 30, 2019

Author Contributor

Seems like private_data_len field is relevant only in rdmacm and not in ucp. Is that correct?

I had this field left as earlier on as sockcm is mostly derived from rdmacm uct. If I understand correctly, what is relevant is the length of the packed addresses that are sent over to the server to pass as the last argument to iface->conn_request_cb (which is represented using hdr->length in the case of rdmacm and now sockcm as well). If yes, then I'll plan to get rid of conn_param.private_data_len altogether in sockcm (because it's not used as such) and just keep hdr->length and use that in the callback instead of conn_param.private_data_len. Lmk if my understanding is right here and if I can proceed. Thanks!

This comment has been minimized.

Copy link
@alinask

alinask Jul 31, 2019

Contributor

yeah, seems that conn_param.private_data_len is redundant.

src/uct/tcp/sockcm/sockcm_ep.c Outdated Show resolved Hide resolved
src/uct/tcp/sockcm/sockcm_md.c Outdated Show resolved Hide resolved
ucs_debug("connect confirm = %d\n", connect_confirm);

sent_len = send(ep->sock_id_ctx->sock_id, (char *) &conn_param + offset,
(conn_param.private_data_len - offset), 0);

This comment has been minimized.

Copy link
@alinask

alinask May 13, 2019

Contributor

maybe send (hdr->length + sizeof(uct_sockcm_priv_data_hdr_t)) as the len?
conn_param.private_data_len can be too long since the actual data written can be shorter.

This comment has been minimized.

This comment has been minimized.

Copy link
@alinask

alinask Jun 5, 2019

Contributor

how is this related..?

@alinask

This comment has been minimized.

Copy link
Contributor

commented May 16, 2019

bot:retest

@swx-jenkins

This comment has been minimized.

Copy link
Collaborator

commented May 16, 2019

@alinask

This comment has been minimized.

Copy link
Contributor

commented May 16, 2019

11:38:47 /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node/worker/3/contrib/../src/uct/tcp/sockcm/sockcm_ep.c:66:21: error: comparison of unsigned expression < 0 is always false [-Werror,-Wtautological-compare]
11:38:47     if (hdr->length < 0) {
11:38:47         ~~~~~~~~~~~ ^ ~
11:38:47 1 error generated.
12:01:56 [----------] 1 test from shm/test_ucp_sockaddr_with_rma_atomic
12:01:56 [ RUN      ] shm/test_ucp_sockaddr_with_rma_atomic.wireup/2
12:01:56 [     INFO ] Testing 65.65.65.0:52845
12:01:57 [     INFO ] < bind(fd=43) failed: Address already in use >
12:01:57 /scrap/jenkins/workspace/hpc-ucx-pr-2/label/vulcan-jenkins/worker/0/contrib/../test/gtest/ucp/ucp_test.cc:593: Failure
12:01:57 Value of: status
12:01:57   Actual: -3
12:01:57 Expected: UCS_ERR_UNREACHABLE
12:01:57 Which is: -6
12:01:57 [     SKIP ] (connect(fd=43, dest_addr=65.65.65.0:52845) failed: Connection refused)
12:01:57 
[  FAILED  ] shm/test_ucp_sockaddr_with_rma_atomic.wireup/2, where GetParam() = shm (664 ms)
11:58:54 [----------] 1 test from tcp/test_uct_peer_failure
11:58:54 [ RUN      ] tcp/test_uct_peer_failure.purge_failed_peer/0
11:58:54 [     INFO ] < send(fd=15 data=0x12cfe00 length=13) failed: Broken pipe >
11:58:54 [     INFO ] < send(fd=15 data=0x12cfe00 length=13) failed: Broken pipe >
11:58:54 [     INFO ] < send(fd=15 data=0x12cfe00 length=13) failed: Broken pipe >
11:58:54 [     INFO ] < send(fd=15 data=0x12cfe00 length=13) failed: Broken pipe >
11:58:54 [     INFO ] < send(fd=15 data=0x12cfe00 length=13) failed: Broken pipe >
12:02:32 [----------] 5 tests from tcp/test_ucp_peer_failure
12:02:32 [ RUN      ] tcp/test_ucp_peer_failure.zcopy/1
12:02:32 [1557997352.255457] [hpc-test-node-upstream:28899:0]     ucp_worker.c:1369 UCX  INFO  ep_cfg[1]: tag(tcp/br0); 
12:17:32 /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-test-node-upstream/worker/0/contrib/../test/gtest/common/test_helpers.cc:42: Failure
12:17:32 Failed
12:17:32 Connection timed out - abort testing
@alinask

This comment has been minimized.

Copy link
Contributor

commented May 16, 2019

12:02:11 [----------] 9 tests from rcx/test_ucp_tag_match
12:02:11 [ RUN      ] rcx/test_ucp_tag_match.send_recv_nb_partial_exp_medium/1
12:02:11 [1557997331.845988] [hpc-arm-hwi-jenkins:46983:0]     ucp_worker.c:1369 UCX  INFO  ep_cfg[1]: tag(rc_mlx5/mlx5_0:1); 
12:17:11 /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/common/test_helpers.cc:42: Failure
12:17:11 Failed
12:17:11 Connection timed out - abort testing
12:17:11 [hpc-arm-hwi-jenkins:46983:0:46983] Caught signal 6 (Aborted: tkill(2) or tgkill(2))
12:17:13 ==== backtrace (tid:  46983) ====
12:17:13  0 0x000000000000edb0 __lll_lock_wait()  :0
12:17:13  1 0x0000000000009a48 __pthread_mutex_lock()  :0
12:17:13  2 0x000000000055ca2c ucs::test_base::count_warns_logger()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/common/test.cc:131
12:17:13  3 0x0000000000051f74 ucs_log_dispatch()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../src/ucs/debug/log.c:193
12:17:13  4 0x0000000000023f38 ucp_worker_print_used_tls()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../src/ucp/core/ucp_worker.c:1369
12:17:13  5 0x0000000000064590 ucp_wireup_init_lanes()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../src/ucp/wireup/wireup.c:767
12:17:13  6 0x0000000000064d50 ucp_wireup_init_lanes_by_request()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../src/ucp/wireup/wireup.c:233
12:17:13  7 0x0000000000066078 ucp_wireup_msg_handler()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../src/ucp/wireup/wireup.c:532
12:17:13  8 0x0000000000062574 uct_iface_invoke_am()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../src/uct/base/uct_iface.h:599
12:17:13  9 0x0000000000069dec uct_ud_mlx5_iface_poll_rx()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../src/uct/ib/ud/accel/ud_mlx5.c:420
12:17:13 10 0x0000000000022764 uct_iface_progress()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../src/uct/api/uct.h:2701
12:17:13 11 0x00000000000474a8 ucs_callbackq_slow_proxy()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../src/ucs/datastruct/callbackq.c:393
12:17:13 12 0x0000000000025754 ucs_callbackq_dispatch()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../src/ucs/datastruct/callbackq.h:211
12:17:13 13 0x000000000077e6fc ucp_test_base::entity::progress()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/ucp/ucp_test.cc:631
12:17:13 14 0x000000000073a09c test_ucp_tag::wait()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/ucp/test_ucp_tag.cc:112
12:17:13 15 0x000000000073a758 test_ucp_tag::send_b()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/ucp/test_ucp_tag.cc:216
12:17:13 16 0x00000000006e7860 test_ucp_tag_match_send_recv_nb_partial_exp_medium_Test::test_body()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/ucp/test_ucp_tag_match.cc:273
12:17:13 17 0x000000000055d49c ucs::test_base::run()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/common/test.cc:276
12:17:13 18 0x000000000054ade4 HandleSehExceptionsInMethodIfSupported<testing::Test, void>()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/common/gtest-all.cc:3562
12:17:13 19 0x000000000053f59c testing::Test::Run()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/common/gtest-all.cc:3635
12:17:13 20 0x000000000053f66c testing::TestInfo::Run()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/common/gtest-all.cc:3812
12:17:13 21 0x000000000053f7dc testing::TestCase::Run()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/common/gtest-all.cc:3930
12:17:13 22 0x0000000000543bec testing::internal::UnitTestImpl::RunAllTests()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/common/gtest-all.cc:5802
12:17:13 23 0x0000000000543f08 testing::internal::UnitTestImpl::RunAllTests()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/common/gtest-all.cc:5719
12:17:13 24 0x00000000004ee5c0 RUN_ALL_TESTS()  /scrap/jenkins/workspace/hpc-ucx-pr/label/hpc-arm-hwi-jenkins/worker/2/contrib/../test/gtest/common/gtest.h:20059
12:17:13 25 0x0000000000020d64 __libc_start_main()  :0
12:17:13 26 0x0000000000529f14 _start()  :0
12:17:13 =================================
12:17:13 Sending notification to akvenkatesh@nvidia.com
12:10:53 [ RUN      ] shm_ib/test_ucp_sockaddr.query_listener/0
12:10:53 [     INFO ] Testing 11.151.71.0:40131
12:10:53 [     INFO ] < bind(fd=48) failed: Address already in use >
12:10:53 /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/ucp/ucp_test.cc:593: Failure
12:10:53 Value of: status
12:10:53   Actual: -3
12:10:53 Expected: UCS_ERR_UNREACHABLE
12:10:53 Which is: -6
12:10:53 [hpc-test-node3:19602:0:19602] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x158)
12:10:53 
12:10:53 /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../src/ucp/core/ucp_listener.c: [ ucp_listener_query() ]
12:10:53       ...
12:10:53       169 ucs_status_t ucp_listener_query(ucp_listener_h listener, ucp_listener_attr_t *attr)
12:10:53       170 {
12:10:53       171     if (attr->field_mask & UCP_LISTENER_ATTR_FIELD_PORT) {
12:10:53 ==>   172         attr->port = listener->wiface.attr.listen_port;
12:10:53       173     }
12:10:53       174 
12:10:53       175     return UCS_OK;
12:10:53 
12:10:54 ==== backtrace (tid:  19602) ====
12:10:54  0 0x000000000001cab5 ucp_listener_query()  /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../src/ucp/core/ucp_listener.c:172
12:10:54  1 0x000000000077b012 test_ucp_sockaddr_query_listener_Test::test_body()  /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/ucp/test_ucp_sockaddr.cc:455
12:10:54  2 0x00000000005523f2 ucs::test_base::run()  /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/common/test.cc:276
12:10:54  3 0x00000000005524ed ucs::test_base::TestBodyProxy()  /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/common/test.cc:302
12:10:54  4 0x000000000053fe9a testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>()  /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/common/gtest-all.cc:3562
12:10:54  5 0x0000000000537f80 testing::Test::Run()  /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/common/gtest-all.cc:3634
12:10:54  6 0x00000000005382e5 testing::Test::Run()  /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/common/gtest-all.cc:3626
12:10:54  7 0x000000000053841d testing::TestInfo::Run()  /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/common/gtest-all.cc:3929
12:10:54  8 0x0000000000538e5d testing::TestCase::Run()  /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/common/gtest-all.cc:5800
12:10:54  9 0x00000000005390d5 testing::internal::UnitTestImpl::RunAllTests()  /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/common/gtest-all.cc:5719
12:10:54 10 0x00000000005210c6 RUN_ALL_TESTS()  /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/common/gtest.h:20059
12:10:54 11 0x0000000000021c05 __libc_start_main()  ???:0
12:10:54 12 0x0000000000521cb8 _start()  ???:0
12:10:54 =================================
12:10:52 [ RUN      ] shm_ib/test_ucp_sockaddr.err_handle/0
12:10:52 [     INFO ] < bind(fd=48) failed: Address already in use >
12:10:52 /scrap/jenkins/workspace/hpc-ucx-pr-2/label/hpc-test-node3/worker/1/contrib/../test/gtest/ucp/ucp_test.cc:593: Failure
12:10:52 Value of: status
12:10:52   Actual: -3
12:10:52 Expected: UCS_ERR_UNREACHABLE
12:10:52 Which is: -6
12:10:52 [     SKIP ] (connect(fd=48, dest_addr=11.151.71.0:256) failed: Connection refused)
12:10:52 [  FAILED  ] shm_ib/test_ucp_sockaddr.err_handle/0, where GetParam() = shm,ib (303 ms)
@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented May 16, 2019

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/10252/ for details (Mellanox internal link).

src/uct/tcp/sockcm/sockcm_iface.c Show resolved Hide resolved
return 0;
}

if (connect(sock_id, addr, addrlen)) {

This comment has been minimized.

Copy link
@alinask

alinask May 19, 2019

Contributor

there is no listen() on the server side for this connect() so the client isn't really connected after connect() returns. so this is probably not the correct way to check reachability for sockcm.

This comment has been minimized.

Copy link
@Akshay-Venkatesh

Akshay-Venkatesh Jun 4, 2019

Author Contributor

If server isn't listening, we get connection refused error (ECONNREFUSED) and this function still returns that the server is reachable should that error be detected. That should suffice for reachability right? If the server can't be reached we'll get back ENETUNREACH error and we return that the addr isn't reachable by returning 0.
https://github.com/openucx/ucx/pull/3570/files#diff-e6d8a43b8d2e3efb16e0f895a16f4486R63

That said, calling connect isn't needed but just calling getnameinfo will tell if the given address is reachable from transports that support sockaddr. getnameinfo will return an error like:

 getnameinfo error : Name or service not known

Which should suffice, right? I'll remove the connect part. Lmk if this is ok.

This comment has been minimized.

Copy link
@alinask

alinask Jun 5, 2019

Contributor

Since connect() can return even though the server didn't call accept() yet, I agree that connect() isn't a good indication here. I'm not sure though about getnameinfo() - can you please check this with gtest?

This comment has been minimized.

Copy link
@Akshay-Venkatesh

Akshay-Venkatesh Jun 5, 2019

Author Contributor

will do

This comment has been minimized.

Copy link
@alinask

alinask Jun 6, 2019

Contributor

10x! Also, please note, that in order to test sockcm in gtest, will need to make sure that both rdmacm and sockcm are tested everywhere - in uct's gtest, in ucp's gtest, and here as well:
https://github.com/openucx/ucx/blob/master/test/gtest/uct/test_md.cc#L523

src/uct/tcp/sockcm/sockcm_ep.c Outdated Show resolved Hide resolved
ucs_warn("sockcm does not support SIGIO");
}

self->sock_id = -1;

This comment has been minimized.

Copy link
@alinask

alinask May 20, 2019

Contributor

where is this sock_id used?

This comment has been minimized.

Copy link
@alinask

alinask Jun 5, 2019

Contributor

In my question I asked about sock_id on the iface:
https://github.com/openucx/ucx/pull/3570/files#diff-ad83c0e9658b99e92394fc8e0fafff1cR25
In the link you're pointing to ep->sock_id_ctx->sock_id:
https://github.com/openucx/ucx/pull/3570/files#diff-02241120d4fc3f8637422e27a8ce100dR39
those are not the same...

This comment has been minimized.

Copy link
@Akshay-Venkatesh

Akshay-Venkatesh Jun 5, 2019

Author Contributor

You're right. I was looking at the wrong one. It seems like this is the only place I use iface->sock_id
https://github.com/openucx/ucx/pull/3570/files#diff-aca878e51b7d5e6eaa75cd20a1375eadR168

Should I remove this if that's the case?

This comment has been minimized.

Copy link
@alinask

alinask Jun 6, 2019

Contributor

if you never use it and never change its value, then yes..

goto out_free;
}

ep->sock_id_ctx->ep = ep;

This comment has been minimized.

Copy link
@alinask

alinask May 20, 2019

Contributor

why is it required to save the ep here?

This comment has been minimized.

Copy link
@Akshay-Venkatesh

Akshay-Venkatesh Jun 4, 2019

Author Contributor

You're right not needed.

This comment has been minimized.

Copy link
@alinask

This comment has been minimized.

Copy link
@Akshay-Venkatesh

Akshay-Venkatesh Jun 5, 2019

Author Contributor

This looks like an older commit. Am I mistaken?

This comment has been minimized.

Copy link
@alinask

alinask Jun 6, 2019

Contributor

the ep field is present in the uct_sockcm_ctx struct in the master branch.

@swx-jenkins3

This comment has been minimized.

Copy link
Collaborator

commented May 22, 2019

Can one of the admins verify this patch?

FNU Akshay Venkatesh
@Akshay-Venkatesh

This comment has been minimized.

Copy link
Contributor Author

commented Jun 4, 2019

@alinask Thanks for the review. Apologies for the late response. I've tried to address most of the concerns you've raised but for two.

I don't have an answer to this: #3570 (comment)
Have a follow-up question on explicitly handling rejection: #3570 (comment)

Lmk your thoughts. Thanks

@swx-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Jun 5, 2019

src/ucs/sys/sock.c Outdated Show resolved Hide resolved
src/uct/tcp/sockcm/sockcm_iface.c Outdated Show resolved Hide resolved
src/uct/tcp/sockcm/sockcm_iface.c Outdated Show resolved Hide resolved
}

static ucs_status_t uct_sockcm_iface_reject(uct_iface_h tl_iface,
uct_conn_request_h conn_request)
{
return UCS_ERR_NOT_IMPLEMENTED;
return UCS_OK;

This comment has been minimized.

Copy link
@alinask

alinask Jun 5, 2019

Contributor

Yes, the client needs to invoke the error handling flow to handle the rejection from the server. Some of the gtests test this.
@evgeny-leksikov

@alinask

This comment has been minimized.

Copy link
Contributor

commented Jun 5, 2019

Hi @Akshay-Venkatesh , I added my comments.
Can you please review the jenkins failures as well?
http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/7583/
Thanks!

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Jun 5, 2019

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/10523/ for details (Mellanox internal link).

@swx-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Jun 5, 2019

@Akshay-Venkatesh

This comment has been minimized.

Copy link
Contributor Author

commented Jun 5, 2019

Hi @Akshay-Venkatesh , I added my comments.
Can you please review the jenkins failures as well?
http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/7583/
Thanks!

Unable to view the link. Can you paste the errors?

@swx-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Aug 16, 2019

@swx-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Aug 16, 2019

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/11943/ for details (Mellanox internal link).

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

Mellanox CI: FAILED on 16 of 29 workers (click for details)

Note: the logs will be deleted after 23-Aug-2019

Agent/Stage Status
_main FAILURE
hpc-arm-cavium-jenkins_W2 FAILURE
hpc-arm-hwi-jenkins_W2 FAILURE
hpc-arm-hwi-jenkins_W3 FAILURE
hpc-test-althca_W1 FAILURE
hpc-test-node-gpu_W0 FAILURE
hpc-test-node-gpu_W1 FAILURE
hpc-test-node-gpu_W3 FAILURE
hpc-test-node-legacy_W0 FAILURE
hpc-test-node-legacy_W1 FAILURE
hpc-test-node-legacy_W3 FAILURE
hpc-test-node-new_W0 FAILURE
hpc-test-node-new_W1 FAILURE
hpc-test-node-new_W2 FAILURE
hpc-test-node-new_W3 FAILURE
r-vmb-ppc-jenkins_W2 FAILURE
hpc-arm-cavium-jenkins_W0 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W1 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W3 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W0 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W1 ✔️ SUCCESS
hpc-test-althca_W0 ✔️ SUCCESS
hpc-test-althca_W2 ✔️ SUCCESS
hpc-test-althca_W3 ✔️ SUCCESS
hpc-test-node-gpu_W2 ✔️ SUCCESS
hpc-test-node-legacy_W2 ✔️ SUCCESS
r-vmb-ppc-jenkins_W0 ✔️ SUCCESS
r-vmb-ppc-jenkins_W1 ✔️ SUCCESS
r-vmb-ppc-jenkins_W3 ✔️ SUCCESS
@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/11944/ for details (Mellanox internal link).

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

Mellanox CI: FAILED on 15 of 29 workers (click for details)

Note: the logs will be deleted after 23-Aug-2019

Agent/Stage Status
_main FAILURE
hpc-arm-cavium-jenkins_W2 FAILURE
hpc-arm-hwi-jenkins_W2 FAILURE
hpc-test-althca_W1 FAILURE
hpc-test-node-gpu_W0 FAILURE
hpc-test-node-gpu_W1 FAILURE
hpc-test-node-gpu_W3 FAILURE
hpc-test-node-legacy_W0 FAILURE
hpc-test-node-legacy_W1 FAILURE
hpc-test-node-legacy_W3 FAILURE
hpc-test-node-new_W0 FAILURE
hpc-test-node-new_W1 FAILURE
hpc-test-node-new_W2 FAILURE
hpc-test-node-new_W3 FAILURE
r-vmb-ppc-jenkins_W2 FAILURE
hpc-arm-cavium-jenkins_W0 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W1 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W3 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W0 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W1 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W3 ✔️ SUCCESS
hpc-test-althca_W0 ✔️ SUCCESS
hpc-test-althca_W2 ✔️ SUCCESS
hpc-test-althca_W3 ✔️ SUCCESS
hpc-test-node-gpu_W2 ✔️ SUCCESS
hpc-test-node-legacy_W2 ✔️ SUCCESS
r-vmb-ppc-jenkins_W0 ✔️ SUCCESS
r-vmb-ppc-jenkins_W1 ✔️ SUCCESS
r-vmb-ppc-jenkins_W3 ✔️ SUCCESS

FNU Akshay Venkatesh added some commits Aug 16, 2019

@swx-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Aug 16, 2019

@swx-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Aug 16, 2019

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/11948/ for details (Mellanox internal link).

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/11947/ for details (Mellanox internal link).

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

Mellanox CI: FAILED on 15 of 29 workers (click for details)

Note: the logs will be deleted after 23-Aug-2019

Agent/Stage Status
_main FAILURE
hpc-arm-cavium-jenkins_W2 FAILURE
hpc-arm-hwi-jenkins_W2 FAILURE
hpc-test-althca_W1 FAILURE
hpc-test-node-gpu_W0 FAILURE
hpc-test-node-gpu_W1 FAILURE
hpc-test-node-gpu_W3 FAILURE
hpc-test-node-legacy_W0 FAILURE
hpc-test-node-legacy_W1 FAILURE
hpc-test-node-legacy_W3 FAILURE
hpc-test-node-new_W0 FAILURE
hpc-test-node-new_W1 FAILURE
hpc-test-node-new_W2 FAILURE
hpc-test-node-new_W3 FAILURE
r-vmb-ppc-jenkins_W2 FAILURE
hpc-arm-cavium-jenkins_W0 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W1 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W3 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W0 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W1 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W3 ✔️ SUCCESS
hpc-test-althca_W0 ✔️ SUCCESS
hpc-test-althca_W2 ✔️ SUCCESS
hpc-test-althca_W3 ✔️ SUCCESS
hpc-test-node-gpu_W2 ✔️ SUCCESS
hpc-test-node-legacy_W2 ✔️ SUCCESS
r-vmb-ppc-jenkins_W0 ✔️ SUCCESS
r-vmb-ppc-jenkins_W1 ✔️ SUCCESS
r-vmb-ppc-jenkins_W3 ✔️ SUCCESS
@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2019

Mellanox CI: FAILED on 15 of 29 workers (click for details)

Note: the logs will be deleted after 23-Aug-2019

Agent/Stage Status
_main FAILURE
hpc-arm-cavium-jenkins_W2 FAILURE
hpc-arm-hwi-jenkins_W2 FAILURE
hpc-test-althca_W1 FAILURE
hpc-test-node-gpu_W0 FAILURE
hpc-test-node-gpu_W1 FAILURE
hpc-test-node-gpu_W3 FAILURE
hpc-test-node-legacy_W0 FAILURE
hpc-test-node-legacy_W1 FAILURE
hpc-test-node-legacy_W3 FAILURE
hpc-test-node-new_W0 FAILURE
hpc-test-node-new_W1 FAILURE
hpc-test-node-new_W2 FAILURE
hpc-test-node-new_W3 FAILURE
r-vmb-ppc-jenkins_W2 FAILURE
hpc-arm-cavium-jenkins_W0 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W1 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W3 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W0 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W1 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W3 ✔️ SUCCESS
hpc-test-althca_W0 ✔️ SUCCESS
hpc-test-althca_W2 ✔️ SUCCESS
hpc-test-althca_W3 ✔️ SUCCESS
hpc-test-node-gpu_W2 ✔️ SUCCESS
hpc-test-node-legacy_W2 ✔️ SUCCESS
r-vmb-ppc-jenkins_W0 ✔️ SUCCESS
r-vmb-ppc-jenkins_W1 ✔️ SUCCESS
r-vmb-ppc-jenkins_W3 ✔️ SUCCESS
FNU Akshay Venkatesh
@swx-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Aug 17, 2019

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 17, 2019

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/11949/ for details (Mellanox internal link).

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 17, 2019

Mellanox CI: FAILED on 15 of 29 workers (click for details)

Note: the logs will be deleted after 24-Aug-2019

Agent/Stage Status
_main FAILURE
hpc-arm-cavium-jenkins_W2 FAILURE
hpc-arm-hwi-jenkins_W2 FAILURE
hpc-test-althca_W1 FAILURE
hpc-test-node-gpu_W0 FAILURE
hpc-test-node-gpu_W1 FAILURE
hpc-test-node-gpu_W3 FAILURE
hpc-test-node-legacy_W0 FAILURE
hpc-test-node-legacy_W1 FAILURE
hpc-test-node-legacy_W3 FAILURE
hpc-test-node-new_W0 FAILURE
hpc-test-node-new_W1 FAILURE
hpc-test-node-new_W2 FAILURE
hpc-test-node-new_W3 FAILURE
r-vmb-ppc-jenkins_W2 FAILURE
hpc-arm-cavium-jenkins_W0 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W1 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W3 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W0 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W1 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W3 ✔️ SUCCESS
hpc-test-althca_W0 ✔️ SUCCESS
hpc-test-althca_W2 ✔️ SUCCESS
hpc-test-althca_W3 ✔️ SUCCESS
hpc-test-node-gpu_W2 ✔️ SUCCESS
hpc-test-node-legacy_W2 ✔️ SUCCESS
r-vmb-ppc-jenkins_W0 ✔️ SUCCESS
r-vmb-ppc-jenkins_W1 ✔️ SUCCESS
r-vmb-ppc-jenkins_W3 ✔️ SUCCESS
@swx-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Aug 17, 2019

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 17, 2019

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/11951/ for details (Mellanox internal link).

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 17, 2019

Mellanox CI: FAILED on 10 of 29 workers (click for details)

Note: the logs will be deleted after 24-Aug-2019

Agent/Stage Status
_main FAILURE
hpc-arm-cavium-jenkins_W2 FAILURE
hpc-arm-hwi-jenkins_W2 FAILURE
hpc-test-althca_W1 FAILURE
hpc-test-node-gpu_W1 FAILURE
hpc-test-node-legacy_W0 FAILURE
hpc-test-node-legacy_W1 FAILURE
hpc-test-node-new_W1 FAILURE
hpc-test-node-new_W2 FAILURE
r-vmb-ppc-jenkins_W2 FAILURE
hpc-arm-cavium-jenkins_W0 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W1 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W3 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W0 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W1 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W3 ✔️ SUCCESS
hpc-test-althca_W0 ✔️ SUCCESS
hpc-test-althca_W2 ✔️ SUCCESS
hpc-test-althca_W3 ✔️ SUCCESS
hpc-test-node-gpu_W0 ✔️ SUCCESS
hpc-test-node-gpu_W2 ✔️ SUCCESS
hpc-test-node-gpu_W3 ✔️ SUCCESS
hpc-test-node-legacy_W2 ✔️ SUCCESS
hpc-test-node-legacy_W3 ✔️ SUCCESS
hpc-test-node-new_W0 ✔️ SUCCESS
hpc-test-node-new_W3 ✔️ SUCCESS
r-vmb-ppc-jenkins_W0 ✔️ SUCCESS
r-vmb-ppc-jenkins_W1 ✔️ SUCCESS
r-vmb-ppc-jenkins_W3 ✔️ SUCCESS
@Akshay-Venkatesh

This comment has been minimized.

Copy link
Contributor Author

commented Aug 17, 2019

@yosefe @alinask @evgeny-leksikov I'm seeing two remaining errors: 1. in sockaddr_accessibility/0 which I don't see locally and which I don't see across all tests. Is it possible to provide TRACE logs for this test? 2. I see some tests fail in CI even though all gtests have passed. Is this because of valgrind errors? Can you point to errors?

Thanks

@swx-jenkins

This comment has been minimized.

Copy link
Collaborator

commented Aug 21, 2019

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2019

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/11996/ for details (Mellanox internal link).

@mellanox-github

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2019

Mellanox CI: FAILED on 6 of 29 workers (click for details)

Note: the logs will be deleted after 28-Aug-2019

Agent/Stage Status
_main FAILURE
hpc-test-althca_W2 FAILURE
hpc-test-node-gpu_W3 FAILURE
hpc-test-node-legacy_W3 FAILURE
hpc-test-node-new_W3 FAILURE
r-vmb-ppc-jenkins_W3 FAILURE
hpc-arm-cavium-jenkins_W0 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W1 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W2 ✔️ SUCCESS
hpc-arm-cavium-jenkins_W3 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W0 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W1 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W2 ✔️ SUCCESS
hpc-arm-hwi-jenkins_W3 ✔️ SUCCESS
hpc-test-althca_W0 ✔️ SUCCESS
hpc-test-althca_W1 ✔️ SUCCESS
hpc-test-althca_W3 ✔️ SUCCESS
hpc-test-node-gpu_W0 ✔️ SUCCESS
hpc-test-node-gpu_W1 ✔️ SUCCESS
hpc-test-node-gpu_W2 ✔️ SUCCESS
hpc-test-node-legacy_W0 ✔️ SUCCESS
hpc-test-node-legacy_W1 ✔️ SUCCESS
hpc-test-node-legacy_W2 ✔️ SUCCESS
hpc-test-node-new_W0 ✔️ SUCCESS
hpc-test-node-new_W1 ✔️ SUCCESS
hpc-test-node-new_W2 ✔️ SUCCESS
r-vmb-ppc-jenkins_W0 ✔️ SUCCESS
r-vmb-ppc-jenkins_W1 ✔️ SUCCESS
r-vmb-ppc-jenkins_W2 ✔️ SUCCESS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.