Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove debconf apt progress #2962

Merged
merged 5 commits into from Jan 4, 2020

Conversation

bcambl
Copy link
Member

@bcambl bcambl commented Oct 13, 2019

By submitting this pull request, I confirm the following:

  • I have read and understood the contributors guide, as well as this entire template.
  • I have made only one major change in my proposed changes.
  • I have commented my proposed changes within the code.
  • I have tested my proposed changes, and have included unit tests where possible.
  • I am willing to help maintain this change if there are issues with it later.
  • I give this submission freely and claim no ownership.
  • It is compatible with the EUPL 1.2 license
  • I have squashed any insignificant commits.

What does this PR aim to accomplish?:

  • Fix a bug where dependencies would not be installed if the debconf-apt-progress command was unavailable.
  • Simplify installer by removing the need to test/install dependencies for debconf-apt-progress

Original issue report #1278 and fix was #1280
Additional discussion on #2892
Replaces my previous PR #2934

How does this PR accomplish the above?:

  • remove usage of debconf-apt-progress
  • remove unused dialog dependency for Debian (commit by @MichaIng)
  • remove unused dialog dependency for Fedora
  • add visual separation for package manager output

What documentation changes (if any) are needed to support this PR?:
n/a

@@ -245,7 +245,7 @@ if is_command apt-get ; then
fi
# Since our install script is so large, we need several other programs to successfully get a machine provisioned
# These programs are stored in an array so they can be looped through later
INSTALLER_DEPS=(apt-utils dialog debconf dhcpcd5 git "${iproute_pkg}" whiptail)
INSTALLER_DEPS=(apt-utils debconf dhcpcd5 git "${iproute_pkg}" whiptail)
Copy link
Contributor

@MichaIng MichaIng Oct 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bcambl
debconf is then not required as well. It is the actual package that provides debconf-apt-progress command.

And apt-utils never was? Sorry separate topic, but as you're already redoing the APT install method 🙂 ...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking the same but I did not want to remove the packages in the event they are required for a particular minimal Debian based distro.
The two packages are available by default in my Debian 10 test environment.

The following commands are provided by the packages:

bcambl@debian10:~$ dpkg-query -L debconf | grep bin
/usr/bin
/usr/bin/debconf
/usr/bin/debconf-apt-progress
/usr/bin/debconf-communicate
/usr/bin/debconf-copydb
/usr/bin/debconf-escape
/usr/bin/debconf-set-selections
/usr/bin/debconf-show
/usr/sbin
/usr/sbin/dpkg-preconfigure
/usr/sbin/dpkg-reconfigure


bcambl@debian10:~$ dpkg-query -L apt-utils | grep bin
/usr/bin
/usr/bin/apt-extracttemplates
/usr/bin/apt-ftparchive
/usr/bin/apt-sortpkgs
bcambl@debian10:~$ 

I would like to hear from the other team members before removing them as well.
Thanks for the review :)

@PromoFaux
Copy link
Member

This creates a lot of extra output, is that intentional? (ignore missing ticks, having terminal issues)

image

image

image

@MichaIng
Copy link
Contributor

MichaIng commented Oct 17, 2019

@PromoFaux
Jep this is the "downside", depending on how you see it. IMO APT installs are not something trivial in every case and this additional output is useful information, especially when it comes to changed "alternatives", updated config files (user choice whether to update or not), packages which were set on hold etc. The used -qq option is already the one that produces least output, so I don't know any way to further reduce it without breaking possible interactive parts.

Indeed it breaks the previous output style of the installer. On DietPi we separate it visually by setting and reverting the terminal colour to \e[90m...APT output...\e[0m, however on some terminals this has no effect (no 4bit colour code support) or might be hard to read.

What would be IMO good, is some additional line before the actual apt-get call:

 [i] Processing APT install(s) for "${installArray[@]}", please wait...

The "please wait..." is especially helpful on slow devices (RPi1/Zero) with disabled APT cache, where the -qq option leads to a possibly long time without any progress indication.

@PromoFaux
Copy link
Member

To my eye, it would be at least less jarring if it was possible to force the additional output to be offset from the screen like so:

  [i] Checking for dhcpcd5 (will be installed)
  [ ] Checking for iproute2
  [ ] Checking for whiptail

  [i] Configuring dhcpcd5 ...
        Selecting previously unselected package dhcpcd5.
        (Reading database ... 29934 files and directories currently installed.)
        Preparing to unpack ... /dhcpcd5.7.1.0.2_amd64.deb
        Unpacking dhcpcd5 (7.1.0-2) ...
        Setting up dhcpcd5 (7.1.0-2) ...
        Created symlink /etc/systemd/system/multi-user.target.wants/dhcpcd.service -> /lib/systemd/system/dhcpcd.service
        Processing triggers for systemd (241-7~deb10u1) ...

  [i] Using Google (ECS)
  [ ] Set IP address to 178.62.59.70
  [ ] Blah blah blah
  [i] Wish I'd copy pasted originally so I didn't have to type out the text in the screenshot

@MichaIng
Copy link
Contributor

@PromoFaux
Impossible AFAIK, you can set font style (colour, bold, italic, ...) for all following external command output (since these are terminal properties) but you cannot control what characters (including spaces/alignment) are printed. Some horizontal line could be as well used to separate apt-get output visually?

@dschaper
Copy link
Member

@MichaIng
Copy link
Contributor

MichaIng commented Oct 17, 2019

@dschaper
Needs testing: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860931#10
Sounds like expected term.log will be missing and might break interactive input similar to how debconf-apt-progress did.

@PromoFaux
Copy link
Member

Just spit-balling... could we not pipe that output into a variable then offset it with a printf or echo back?

@dschaper
Copy link
Member

It would be a huge amount of text to buffer in memory...

@MichaIng
Copy link
Contributor

MichaIng commented Oct 17, 2019

I guess when redirecting STDOUT and printing back, interactive input fields are overridden 🤔. If user then hits return, default choice is used instead of question + input field being printed again.

@PromoFaux
Copy link
Member

Fair comments both.

If the best we can do is a horizontal line, maybe with a different text colour then it may just look a bit better. Personal preference I guess.. I am just not a fan of a messy output, regardless of how many people will actually end up paying attention to it!

  [i] Checking for dhcpcd5 (will be installed)
  [ ] Checking for iproute2
  [ ] Checking for whiptail

  [i] Output from apt ...
----------------------------------------------------------------------------------------------------------------
Selecting previously unselected package dhcpcd5.
(Reading database ... 29934 files and directories currently installed.)
Preparing to unpack ... /dhcpcd5.7.1.0.2_amd64.deb
Unpacking dhcpcd5 (7.1.0-2) ...
Setting up dhcpcd5 (7.1.0-2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/dhcpcd.service -> /lib/systemd/system/dhcpcd.service
Processing triggers for systemd (241-7~deb10u1) ...
----------------------------------------------------------------------------------------------------------------

  [i] Using Google (ECS)
  [ ] Set IP address to 178.62.59.70
  [ ] Blah blah blah
  [i] Wish I'd copy pasted originally so I didn't have to type out the text in the screenshot

@MichaIng
Copy link
Contributor

[i] Output from apt ...

=>

[i] Processing APT install(s) for "${installArray[@]}", please wait...

due to possible delay until first output, as mentioned above?

@PromoFaux
Copy link
Member

Ah yeah, for sure. We used to have a fancy spinner thing for those situtations, but it was buggy as all hell. I'd hate to dig that out again.

@MichaIng
Copy link
Contributor

Here is ours: https://github.com/MichaIng/DietPi/blob/dev/dietpi/func/dietpi-globals#L329-L379
However we never use it for APT calls due to same issue: If APT output is not redirected, it would mess with animation, and this again breaks interactive input.

Will do some tests with -o=Dpkg::Use-Pty=0 the next days.

@MichaIng
Copy link
Contributor

MichaIng commented Nov 7, 2019

Would love to see this implemented soon.

Did some more tests and sadly couldn't find a way to further reduce APT output while keeping the ability for interactive choices if e.g. config files are about to be updated:

root@VM-Buster:~# apt-get -qq install net-tools
Selecting previously unselected package net-tools.
(Reading database ... 21150 files and directories currently installed.)
Preparing to unpack .../net-tools_1.60+git20180626.aebd88e-1_amd64.deb ...
Unpacking net-tools (1.60+git20180626.aebd88e-1) ...
Setting up net-tools (1.60+git20180626.aebd88e-1) ...
root@VM-Buster:~# DEBIAN_FRONTEND=noninteractive apt-get -qq -o=Dpkg::Use-Pty=0 install net-tools
Selecting previously unselected package net-tools.
(Reading database ... 21150 files and directories currently installed.)
Preparing to unpack .../net-tools_1.60+git20180626.aebd88e-1_amd64.deb ...
Unpacking net-tools (1.60+git20180626.aebd88e-1) ...
Setting up net-tools (1.60+git20180626.aebd88e-1) ...

It seems that -qq already inherits all the other options. DEBIAN_FRONTEND=noninteractive btw does not make it non-interactive... On my personal home server I use force-confask as dpkg.cfg setting (I want to know about every maintainers config update) and DEBIAN_FRONTEND=noninteractive does neither prevent the choice prompt nor input.

@bcambl @dschaper
Do you agree to go with the above and separate the output from other installer output by:

   [i] Processing APT install(s) for "${installArray[@]}", please wait...
-------------------------------------------------------------------------
Selecting previously unselected package net-tools.
(Reading database ... 21150 files and directories currently installed.)
Preparing to unpack .../net-tools_1.60+git20180626.aebd88e-1_amd64.deb ...
Unpacking net-tools (1.60+git20180626.aebd88e-1) ...
Setting up net-tools (1.60+git20180626.aebd88e-1) ...
-------------------------------------------------------------------------

Removing debconf and apt-utils from dependencies can be done as a separate PR but since it somehow belongs together, see: #2962 (comment)

@bcambl
Copy link
Member Author

bcambl commented Dec 1, 2019

@dschaper @PromoFaux have either of you put any additional thought into this?
I do not want to push any more commits here until the desired stdout format has been decided on.

@PromoFaux
Copy link
Member

I think, as long as we can separate it somehow from the "pretty" formatted install output, then we should be good.

I don't really have any suggestions beyond:
#2962 (comment)

@dschaper
Copy link
Member

dschaper commented Dec 1, 2019

Function beats form for me. I know running apt-get in a docker image build is really quiet due to the readline, but I just don't know offhand if readline allows for apt to handle conflicts interactively.

For now it may be ugly but it works and it solves a bug that is often hard to diagnose. We can pretty it up later, fix it in post.

@MichaIng
Copy link
Contributor

MichaIng commented Dec 4, 2019

Definitely function > form. However to have outputs separated clearly, the following could be done here: https://github.com/pi-hole/pi-hole/blob/7052a7695e5b25e20d488bffd2fc94595e90a2e1/automated%20install/basic-install.sh#L1637
Replace this line by:

  printf "  %b Processing APT install(s) for %s, please wait...\\n" "${INFO}": "${installArray[@]}"
  echo '-------------------------------------------------------------------------'
  "${PKG_INSTALL[@]}" "${installArray[@]}"
  echo '-------------------------------------------------------------------------'

@@ -1652,7 +1652,7 @@ install_dependent_packages() {
fi
done
if [[ "${#installArray[@]}" -gt 0 ]]; then
"${PKG_INSTALL[@]}" "${installArray[@]}" &> /dev/null
Copy link
Contributor

@MichaIng MichaIng Dec 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bcambl
Not sure if this is wanted, since this is a yum install, not an apt-get install! Not sure if yum can hang for the same reason, when redirecting STDOUT, but in all other yum calls &> /dev/null is still in place, hence this makes it inconsistent. I suggest to revert it in this PR. What we've done for apt-get could be done for yum as well (if required for same reason), but should be a dedicated PR then, where I would change PKG_INSTALL to be a function to handle apt-get + yum/dnf all together in a consistent way.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I was to keep this /dev/null in place, there would be no output between the horizontal rule on yum/dnf distributions. As for the redirections in other locations...I think they are fine for now.
If we decide to remove them, those changes would belong in a different PR.

@MichaIng
Copy link
Contributor

MichaIng commented Dec 11, 2019

To push it a bid @reviewer(s) @dschaper:

   [i] Processing APT install(s) for net-tools, please wait...
-------------------------------------------------------------------------
Selecting previously unselected package net-tools.
(Reading database ... 21150 files and directories currently installed.)
Preparing to unpack .../net-tools_1.60+git20180626.aebd88e-1_amd64.deb ...
Unpacking net-tools (1.60+git20180626.aebd88e-1) ...
Setting up net-tools (1.60+git20180626.aebd88e-1) ...
-------------------------------------------------------------------------

@PromoFaux
Copy link
Member

Do you agree with: #2962 (comment)

Yep, If that's the best we can do to separate the output, I'm good with it. Other developer's opinions may vary

@bcambl
Copy link
Member Author

bcambl commented Dec 12, 2019

Thanks all, I will update this PR soon.

MichaIng and others added 2 commits January 1, 2020 13:11
+ The installer uses `whiptail`, thus `dialog` is not required.

Signed-off-by: MichaIng <micha@dietpi.com>
Signed-off-by: bcambl <blayne@blaynecampbell.com>
…es()

Removes the need for conditional debconf-apt-progress dependency checking

Signed-off-by: bcambl <blayne@blaynecampbell.com>
Signed-off-by: bcambl <blayne@blaynecampbell.com>
Signed-off-by: bcambl <blayne@blaynecampbell.com>
@bcambl bcambl force-pushed the remove_debconf-apt-progress branch from 7052a76 to 60c5188 Compare January 1, 2020 19:26
@bcambl
Copy link
Member Author

bcambl commented Jan 1, 2020

I have rebased on development.

@bcambl
Copy link
Member Author

bcambl commented Jan 1, 2020

As for commit: 60c5188

The two debian utilities (apt-utils & debconf) were removed as suggested by @MichaIng.

@bcambl bcambl mentioned this pull request Jan 2, 2020
8 tasks
@PromoFaux PromoFaux self-requested a review January 2, 2020 21:05
Copy link
Member

@PromoFaux PromoFaux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed works as described, and the horizontal rule goes some distance to separating the apt output from the installer output. Still a lot more verbose and jarring than I would like, but that's just my opinion, and I can see the need for doing it this way :)

@dschaper
Copy link
Member

dschaper commented Jan 3, 2020

Agreed with Promo about the verbosity but perfect is the enemy of good at this point.

@PromoFaux PromoFaux merged commit 574f7c1 into pi-hole:development Jan 4, 2020
@bcambl bcambl deleted the remove_debconf-apt-progress branch January 4, 2020 16:07
@pralor-bot
Copy link

This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/pihole-funktioniert-nicht-mehr-nach-update/32375/32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants