Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deadlock when build docker-sonic-vs.gz #15722

Closed
k-v1 opened this issue Jul 5, 2023 · 3 comments · Fixed by #15735
Closed

Deadlock when build docker-sonic-vs.gz #15722

k-v1 opened this issue Jul 5, 2023 · 3 comments · Fixed by #15735
Assignees
@liushilongbuaa
Labels
MSFT Triaged this issue has been triaged

Comments

@k-v1
Copy link
Contributor

k-v1 commented Jul 5, 2023 •
edited

Description

master branch

apt_installation_lock doesn't work correctly when we install some deb packages (e.g. openssh-server for docker-sonic-vs)

Steps to reproduce the issue:

  1. make configure PLATFORM=vs
  2. make target/docker-sonic-vs.gz
  3. open target/docker-sonic-vs.gz.log

Describe the results you received:

Elapsed time is 17 min.
From target/docker-sonic-vs.gz.log (it takes 10 min):

^[[91mWaiting dpkg lock for 10, 1/60, info: Locked by command: /usr/bin/apt-get install -y net-tools arping ndisc6 ethtool tcpdump ifupdown bridge-utils python-ply libqt5core5a libqt5network5 libgmp10 libju     dydebian1 openssh-client openssh-server libc-ares2 iproute2 grub2-common bash-completion libelf1 libmnl0 logrotate apt-utils psmisc python3-scapy conntrack iptables jq libzmq5 libzmq3-dev uuid-dev build-ess     ential python3-dev libssl-dev swig openssl libcairo2-dev libdbus-1-dev libgirepository1.0-dev libsystemd-dev pkg-config gir1.2-glib-2.0 libdbus-1-3 libgirepository-1.0-1 dbus redis-server

 ^[[0m^[[91mWaiting dpkg lock for 10, 2/60, info: Locked by command: /usr/bin/apt-get install -y net-tools arping ndisc6 ethtool tcpdump ifupdown bridge-utils python-ply libqt5core5a libqt5network5 libgmp10      libjudydebian1 openssh-client openssh-server libc-ares2 iproute2 grub2-common bash-completion libelf1 libmnl0 logrotate apt-utils psmisc python3-scapy conntrack iptables jq libzmq5 libzmq3-dev uuid-dev buil     d-essential python3-dev libssl-dev swig openssl libcairo2-dev libdbus-1-dev libgirepository1.0-dev libsystemd-dev pkg-config gir1.2-glib-2.0 libdbus-1-3 libgirepository-1.0-1 dbus redis-server

That's because apt-get calls dpkg when install openssh-server:
/usr/bin/dpkg --compare-versions lt-nl 1:6.6p1-1

Describe the results you expected:

Elapsed time is 3-5 min and no errors in build log.

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):
@k-v1
Copy link
Contributor Author

k-v1 commented Jul 5, 2023

@liushilongbuaa
Could you help with this issue?
You already fixed #13150 when dpkg calls dpkg. But as you can see, apt-get also can call dpkg and we need something to avoid lock loop in this case.

@k-v1
Copy link
Contributor Author

k-v1 commented Jul 5, 2023

One possible solution is to lock only when we install or uninstall a package using dpkg and to ignore other actions (like --compare-versions). But we need to check that we don't break anything else in this case.

@k-v1 k-v1 mentioned this issue Jul 7, 2023
Merged
10 tasks
@vmittal-msft vmittal-msft added Triaged this issue has been triaged MSFT labels Jul 19, 2023
@liushilongbuaa
Copy link
Contributor

The issue is true. I will check the fix PR.

lguohan pushed a commit that referenced this issue May 19, 2024
@k-v1
[Build]: Fix dpkg lock (#15735)
60b1440 
Fix #15722

We don't need to use a lock file every time we call dpkg.
For some commands, like dpkg --print-architecture or dpkg --compare-versions, this action is not required.
Moreover, sometimes these commands are called by apt-get and dpkg -i, and we get a deadlock in this case.

How I did it
Use dpkg lock only for dpkg commands that use /var/lib/dpkg/lock or /var/lib/dpkg/lock-frontend.
I mean dpkg -i, dpkg -P, ...

How to verify it
Check there is no dpkg lock errors in build log.
Check build time of docker-sonic-vs.gz:

old:
08:57:14[ building ] [ target/docker-sonic-vs.gz ]
09:12:47 [ finished ] [ target/docker-sonic-vs.gz ]

new:
01:45:12[ building ] [ target/docker-sonic-vs.gz ]
01:50:39 [ finished ] [ target/docker-sonic-vs.gz ]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liushilongbuaa liushilongbuaa

Labels
MSFT Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix dpkg lock k-v1/sonic-buildimage
3 participants
@liushilongbuaa @vmittal-msft @k-v1