FRR-10.1 MGMTD lock issue

[jayshankarnair]

I have compiled and installed frr-10.1 in my network element.
Send vtysh command to provision my NE.
I have observed some issue in mgmtd transaction. The default commit timeout in
MGMTD_TXN_CFG_COMMIT_MAX_DELAY_SEC is 600.
I changed it to 20 sec. The reason i am getting lock issue and transaction would get stuck.

Logs and discussion

Mar 06 03:25:11 mgmtd[2512]: [S1P1V-R9RVY] Config Request 4: Datastore ID: 2, XPath: /frr-filter:lib/prefix-list[type='ipv4'][name='FILTER_PRIVATE_NETWORK']/entry[sequence='1']/ipv4-prefix-length-greater-or-equal, Value: (null) Type 2
Mar 06 03:25:11 mgmtd[2512]: [S1P1V-R9RVY] Config Request 5: Datastore ID: 2, XPath: /frr-filter:lib/prefix-list[type='ipv4'][name='FILTER_PRIVATE_NETWORK']/entry[sequence='1']/ipv4-prefix-length-lesser-or-equal, Value: (null) Type 2
Mar 06 03:25:11 mgmtd[2512]: [S1P1V-R9RVY] Config Request 6: Datastore ID: 2, XPath: /frr-filter:lib/prefix-list[type='ipv4'][name='FILTER_PRIVATE_NETWORK']/entry[sequence='1']/any, Value: (null) Type 2
Mar 06 03:25:11 mgmtd[2512]: [X3G8F-PM93W] BE-adapter: mgmt_msg_read: got EOF/disconnect
Mar 06 03:25:11 mgmtd[2512]: [ZKV4M-E24MG] mgmt_txn_notify_be_adapter_conn start
Mar 06 03:25:11 mgmtd[2512]: [H905J-MEJKS] mgmt_txn_req_free
Mar 06 03:25:11 mgmtd[2512]: [H905J-MEJKS] mgmt_txn_cfg_batch_free
Mar 06 03:25:11 mgmtd[2512]: [YKHB9-ND03T] BE-adapter: msg_conn_send_msg: can't send message on closed connection
Mar 06 03:25:11 mgmtd[2512]: [NVXX9-7RV6V] mgmt_txn_notify_be_adapter_conn end
Mar 06 03:25:11 mgmtd[2512]: [H905J-MEJKS] mgmt_txn_delete
Mar 06 03:25:11 mgmtd[2512]: [GGJTQ-VTT01] SET_CONFIG request for client 0x2 failed, Error: 'Backend daemon disconnected while processing commit!'

--->Mar 06 03:25:12 mgmtd[2512]: [WEJ55-ZGM1W] Locking for DS 1 failed, Err: 'Lock already taken on DS by another session!' vty 0x559d885cc060
--->Mar 06 03:25:31 mgmtd[2512]: [H7JX4-WT6FK] mgmt_txn_cfg_commit_timedout: ERROR: Backend timeout txn-id: 20 aborting commit

How should i handle the lock error.
1> Wait for the timeout and reissue the command.
2> There is issue with the particular command send on vtysh. Check the command
3> If i change the commit timeout to 20 sec, would it affect the transaction. Provisioning may not be done??.

Looking forward for reply to resolve the issue.

[choppsv1]

It seems like maybe something crashed when doing the initial config? Can you check for that?

[choppsv1]

ping -- do you have full logs from the run

[jayshankarnair]

Attach logs

lock_log_mgmtd.zip.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.

[frrbot]

This issue will be automatically closed in the specified period unless there is further activity.

[choppsv1]

The logs do indicate that client 0x2 disconnected before another config came in and it looks like locks were not cleaned up properly. This lock and cleanup code is known to be incorrect in versions prior to the current master.

Is it possible for you to try running your test using the current code on master? I'm going to be pushing to also get these changes backported to 10.5 branch but that hasn't happened yet.