Debian Template: network/00notiy-hook slows down package installation

[adrelanos]

https://github.com/QubesOS/qubes-core-agent-linux/blob/bd9a3bf515a2139a7232a1d8ac785016bed8e425/network/00notiy-hook runs each and every time when dpkg exited. And dpkg is executed very often by apt-get.

For my better explanation, what is it good for?

Shouldn't the name be notify rather than notiy?

What you want is probably want is an apt-get hook. Not a dpkg hook, right?

Somehow you can also run triggers at the end of apt-get. Such as updating man-db is something that is invoked just once at the end of a run of apt-get. Not each and very package calls this. Individual packges only set the trigger and apt-get invokes the required action at the end when appropriate. We should try to implement it that way.

[marmarek]

+1 for that trigger way. Also we could restore actually checking if all
the updates are installed then.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

+1 for that trigger way. Also we could restore actually checking if all
the updates are installed then.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[nrgaway]

On 14 July 2015 at 11:21, Patrick Schleizer notifications@github.com
wrote:

https://github.com/QubesOS/qubes-core-agent-linux/blob/bd9a3bf515a2139a7232a1d8ac785016bed8e425/network/00notiy-hook
runs each and every time when dpkg exited. And dpkg is executed very often
by apt-get.

Yes it is run many times, but it should not slow anything down as it is
only sending an echo. Other hooks like packagekit are more likely to
slow things down since they may update their cache.

For my better explanation, what is it good for?

This is used to notify dom0 that an update have been preformed so
qubes-manager will remove the updates are available icon for the
template.

It was the only way I could find to run an apt-get update after apt-get
at the time.

Shouldn't the name be notify rather than notiy?

Yes.

What you want is probably want is an apt-get hook. Not a dpkg hook, right?

Yes. I was thinking of changing 00notify-hook (notiy-hook) it to use:

APT::Update::Post-Invoke-Success{
"/usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c 'echo
0' || true";
};

Somehow you can also run triggers at the end of apt-get. Such as updating
man-db is something that is invoked just once at the end of a run of
apt-get. Not each and very package calls this. Individual packges only set
the trigger and apt-get invokes the required action at the end when
appropriate. We should try to implement it that way.

How does the method I suggested above sound?

[nrgaway]

On 14 July 2015 at 11:21, Patrick Schleizer notifications@github.com
wrote:

https://github.com/QubesOS/qubes-core-agent-linux/blob/bd9a3bf515a2139a7232a1d8ac785016bed8e425/network/00notiy-hook
runs each and every time when dpkg exited. And dpkg is executed very often
by apt-get.

Yes it is run many times, but it should not slow anything down as it is
only sending an echo. Other hooks like packagekit are more likely to
slow things down since they may update their cache.

For my better explanation, what is it good for?

This is used to notify dom0 that an update have been preformed so
qubes-manager will remove the updates are available icon for the
template.

It was the only way I could find to run an apt-get update after apt-get
at the time.

Shouldn't the name be notify rather than notiy?

Yes.

What you want is probably want is an apt-get hook. Not a dpkg hook, right?

Yes. I was thinking of changing 00notify-hook (notiy-hook) it to use:

APT::Update::Post-Invoke-Success{
"/usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c 'echo
0' || true";
};

Somehow you can also run triggers at the end of apt-get. Such as updating
man-db is something that is invoked just once at the end of a run of
apt-get. Not each and very package calls this. Individual packges only set
the trigger and apt-get invokes the required action at the end when
appropriate. We should try to implement it that way.

How does the method I suggested above sound?

[adrelanos]

Just now tested this. /etc/apt/apt.conf.d/50_test

APT::Update::Post-Invoke-Success { "echo WORKS!"; };

Works. It's a useful hook. Undocumented, looks like. Runs after 'apt-get update' "succeeded".

Unexpected security bonus: the hook is not dispatched in case of DNS resolution for [the security repository (just now tested)] fails. [could be due to an active man-in-the-middle attack]

This is used to notify dom0 that an update have been preformed so qubes-manager will remove the updates are available icon for the template.

updates are available icon for the template is supposed to indicate, that updates are available, that the user is supposed to install, right? Now, running "apt-get update" successfully only means, that package lists have been successfully refreshed. It does not mean, that updates have been successfully applied. Another step is required for that, that is "apt-get dist-upgrade".

To find out [using a script], if all upgrades have been successfully installed, one method I know is running "apt-get --simulate dist-upgrade". This is what whonxicheck does. You can refer to it as an example implementation - hopefully being a secure upgrade notifier - showing how the parsing and checking could work:
https://github.com/Whonix/whonixcheck/blob/dc89041aef00b64e65af59182efcc971cb42d472/usr/lib/whonixcheck/check_operating_system#L32-L70

I think the correct thing to do is running such a (apt-get --simulate dist-upgrade) style check before the updates are available icon for the template should be removed.

[adrelanos]

Just now tested this. /etc/apt/apt.conf.d/50_test

APT::Update::Post-Invoke-Success { "echo WORKS!"; };

Works. It's a useful hook. Undocumented, looks like. Runs after 'apt-get update' "succeeded".

Unexpected security bonus: the hook is not dispatched in case of DNS resolution for [the security repository (just now tested)] fails. [could be due to an active man-in-the-middle attack]

This is used to notify dom0 that an update have been preformed so qubes-manager will remove the updates are available icon for the template.

updates are available icon for the template is supposed to indicate, that updates are available, that the user is supposed to install, right? Now, running "apt-get update" successfully only means, that package lists have been successfully refreshed. It does not mean, that updates have been successfully applied. Another step is required for that, that is "apt-get dist-upgrade".

To find out [using a script], if all upgrades have been successfully installed, one method I know is running "apt-get --simulate dist-upgrade". This is what whonxicheck does. You can refer to it as an example implementation - hopefully being a secure upgrade notifier - showing how the parsing and checking could work:
https://github.com/Whonix/whonixcheck/blob/dc89041aef00b64e65af59182efcc971cb42d472/usr/lib/whonixcheck/check_operating_system#L32-L70

I think the correct thing to do is running such a (apt-get --simulate dist-upgrade) style check before the updates are available icon for the template should be removed.

[adrelanos]

This caused a whonix-gw-experimental VM to keep apt-get locked for several minutes.

ps aux | grep apt
root      2939  0.0  0.1  18936   688 ?        SLs  14:18   0:00 /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root      2940  0.0  0.0   4192   500 ?        S    14:18   0:00 sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root      2941  0.0  0.7  24132  4496 ?        S    14:18   0:00 /usr/bin/apt-get.anondist-orig -q update
root      2958  0.0  0.6  23600  3404 ?        S    14:18   0:00 /usr/lib/apt/methods/http
root      2959  0.0  0.5  23600  3304 ?        S    14:18   0:00 /usr/lib/apt/methods/http
root      2960  0.0  0.6  23600  3416 ?        S    14:18   0:00 /usr/lib/apt/methods/http
root      2961  0.4  0.6  23600  3636 ?        S    14:18   0:01 /usr/lib/apt/methods/http
root      2962  0.0  0.6  23600  3780 ?        S    14:18   0:00 /usr/lib/apt/methods/http
root      3029  0.0  0.6  23564  3388 ?        S    14:18   0:00 /usr/lib/apt/methods/gpgv
root      3036  0.0  0.6  23552  3448 ?        S    14:18   0:00 /usr/lib/apt/methods/copy
root      3038  0.0  0.5  23552  3336 ?        S    14:18   0:00 /usr/lib/apt/methods/gzip
root      3045  0.0  0.9  27160  5380 ?        S    14:18   0:00 /usr/lib/apt/methods/bzip2
root      3117  0.7  0.6  23552  3596 ?        S    14:21   0:00 /usr/lib/apt/methods/xz
user      3145  0.0  0.3   7844  1992 pts/0    S+   14:22   0:00 grep --color=auto apt

[adrelanos]

This caused a whonix-gw-experimental VM to keep apt-get locked for several minutes.

ps aux | grep apt
root      2939  0.0  0.1  18936   688 ?        SLs  14:18   0:00 /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root      2940  0.0  0.0   4192   500 ?        S    14:18   0:00 sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root      2941  0.0  0.7  24132  4496 ?        S    14:18   0:00 /usr/bin/apt-get.anondist-orig -q update
root      2958  0.0  0.6  23600  3404 ?        S    14:18   0:00 /usr/lib/apt/methods/http
root      2959  0.0  0.5  23600  3304 ?        S    14:18   0:00 /usr/lib/apt/methods/http
root      2960  0.0  0.6  23600  3416 ?        S    14:18   0:00 /usr/lib/apt/methods/http
root      2961  0.4  0.6  23600  3636 ?        S    14:18   0:01 /usr/lib/apt/methods/http
root      2962  0.0  0.6  23600  3780 ?        S    14:18   0:00 /usr/lib/apt/methods/http
root      3029  0.0  0.6  23564  3388 ?        S    14:18   0:00 /usr/lib/apt/methods/gpgv
root      3036  0.0  0.6  23552  3448 ?        S    14:18   0:00 /usr/lib/apt/methods/copy
root      3038  0.0  0.5  23552  3336 ?        S    14:18   0:00 /usr/lib/apt/methods/gzip
root      3045  0.0  0.9  27160  5380 ?        S    14:18   0:00 /usr/lib/apt/methods/bzip2
root      3117  0.7  0.6  23552  3596 ?        S    14:21   0:00 /usr/lib/apt/methods/xz
user      3145  0.0  0.3   7844  1992 pts/0    S+   14:22   0:00 grep --color=auto apt

[adrelanos]

Since this hook is totally broken, i.e. conceptually non-functional and causing issues, here is a pull request to disable it in meanwhile:
https://github.com/marmarek/qubes-core-agent-linux/pull/14

I would appreciate having this hook end up in the new package release, so it won't end up in the new Whonix templates.

Keeping this ticket open, since I guess you want this functionality implemented.

[adrelanos]

Since this hook is totally broken, i.e. conceptually non-functional and causing issues, here is a pull request to disable it in meanwhile:
https://github.com/marmarek/qubes-core-agent-linux/pull/14

I would appreciate having this hook end up in the new package release, so it won't end up in the new Whonix templates.

Keeping this ticket open, since I guess you want this functionality implemented.

[marmarek]

The code you've commented out is already a new version of this hook, which doesn't call apt-get. On process list above you still have the old one, which counts available updates each time. So the current code is already much improved regarding performance. At cost of some functionality (will report that all updates were installed, even when only some of them got installed, or even some unrelated package - in TemplateVM).

[marmarek]

The code you've commented out is already a new version of this hook, which doesn't call apt-get. On process list above you still have the old one, which counts available updates each time. So the current code is already much improved regarding performance. At cost of some functionality (will report that all updates were installed, even when only some of them got installed, or even some unrelated package - in TemplateVM).

[adrelanos]

Also the current implementation is broken. Perhaps no performance issue. But running /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates when dpkg successfully installed a package seems broken by concept to me. I don't see how dpkg installation of a package is related to dom0 update status.

[adrelanos]

Also the current implementation is broken. Perhaps no performance issue. But running /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates when dpkg successfully installed a package seems broken by concept to me. I don't see how dpkg installation of a package is related to dom0 update status.

[marmarek]

Just a clarification: this hook is used to notify dom0, when VM updates
got installed. Just to turn off "updates pending" flag reported in Qubes
Manager. It have nothing to do with updates for dom0.

This is some estimation when updates for template are installed. Far
from ideal one.
What we need is:

1. Some way to run the hook after the whole apt-get call (install,
   upgrade etc), once, or
2. *fast way for checking if all updates are installed.

The first one preferred, of course. Do you have some idea how this can
be done in Debian? Or maybe some other place for such check/notification?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Just a clarification: this hook is used to notify dom0, when VM updates
got installed. Just to turn off "updates pending" flag reported in Qubes
Manager. It have nothing to do with updates for dom0.

This is some estimation when updates for template are installed. Far
from ideal one.
What we need is:

1. Some way to run the hook after the whole apt-get call (install,
   upgrade etc), once, or
2. *fast way for checking if all updates are installed.

The first one preferred, of course. Do you have some idea how this can
be done in Debian? Or maybe some other place for such check/notification?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

Maybe packagekit could do? I'll ask.

I don't know if there is an apt-get "all upgrades have just been installed" hock. I'll ask.

An "on demand" way to check if all upgrades are installed is the following. Please tell if that is fast enough.

To find out [using a script], if all upgrades have been successfully installed, one method I know is running "apt-get --simulate dist-upgrade". This is what whonxicheck does. You can refer to it as an example implementation - hopefully being a secure upgrade notifier - showing how the parsing and checking could work:
https://github.com/Whonix/whonixcheck/blob/dc89041aef00b64e65af59182efcc971cb42d472/usr/lib/whonixcheck/check_operating_system#L32-L70

[adrelanos]

Maybe packagekit could do? I'll ask.

I don't know if there is an apt-get "all upgrades have just been installed" hock. I'll ask.

An "on demand" way to check if all upgrades are installed is the following. Please tell if that is fast enough.

To find out [using a script], if all upgrades have been successfully installed, one method I know is running "apt-get --simulate dist-upgrade". This is what whonxicheck does. You can refer to it as an example implementation - hopefully being a secure upgrade notifier - showing how the parsing and checking could work:
https://github.com/Whonix/whonixcheck/blob/dc89041aef00b64e65af59182efcc971cb42d472/usr/lib/whonixcheck/check_operating_system#L32-L70

[adrelanos]

packagekit: Had a quick lock at the API. Most likely could be used to implement this. I think that, because Apper, which is based on packagekit can find out how many upgrades are available. However, I guess a packagekit based solution would require non-negligible development effort. There is python-packagekit, installable from Debian repository if anyone wants to look into it.

[adrelanos]

packagekit: Had a quick lock at the API. Most likely could be used to implement this. I think that, because Apper, which is based on packagekit can find out how many upgrades are available. However, I guess a packagekit based solution would require non-negligible development effort. There is python-packagekit, installable from Debian repository if anyone wants to look into it.

[marmarek]

On Thu, Aug 06, 2015 at 02:48:04PM -0700, Patrick Schleizer wrote:

An "on demand" way to check if all upgrades are installed is the following. Please tell if that is fast enough.

This is exactly what previous hook was doing. But for being useful, it
needs to be called once, at the end of the whole update process, not
after each package. Otherwise it is way to slow (this is why you've
created this ticket, right? ;) ).

On Thu, Aug 06, 2015 at 02:59:36PM -0700, Patrick Schleizer wrote:

packagekit: Had a quick lock at the API. Most likely could be used to implement this. I think that, because Apper, which is based on packagekit can find out how many upgrades are available. However, I guess a packagekit based solution would require non-negligible development effort. There is python-packagekit, installable from Debian repository if anyone wants to look into it.

This would require running all the updates through PackageKit, right?
Otherwise I guess it will not have any way to know when the packages
were installed.

The main problem is not "how" to check that, but "when". We need some
method to check that between "updates are installed" and "user shut down
the template". That may not be long time, so a polling approach may not
work (and would be rather hacky). Because we haven't found a way for
doing it only once, after updates were installed, we've accepted some
temporary solution for that (running at each dpkg invocation).

Ok, so for now (Whonix 11 templates) I see three options:

1. We find a way to implement it properly.
2. We stay with current solution accepting some inaccuracy in that
   reporting
3. We disable all the "updates pending" notification for Debian-based
   templates (not only Whonix ones).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Thu, Aug 06, 2015 at 02:48:04PM -0700, Patrick Schleizer wrote:

An "on demand" way to check if all upgrades are installed is the following. Please tell if that is fast enough.

This is exactly what previous hook was doing. But for being useful, it
needs to be called once, at the end of the whole update process, not
after each package. Otherwise it is way to slow (this is why you've
created this ticket, right? ;) ).

On Thu, Aug 06, 2015 at 02:59:36PM -0700, Patrick Schleizer wrote:

packagekit: Had a quick lock at the API. Most likely could be used to implement this. I think that, because Apper, which is based on packagekit can find out how many upgrades are available. However, I guess a packagekit based solution would require non-negligible development effort. There is python-packagekit, installable from Debian repository if anyone wants to look into it.

This would require running all the updates through PackageKit, right?
Otherwise I guess it will not have any way to know when the packages
were installed.

The main problem is not "how" to check that, but "when". We need some
method to check that between "updates are installed" and "user shut down
the template". That may not be long time, so a polling approach may not
work (and would be rather hacky). Because we haven't found a way for
doing it only once, after updates were installed, we've accepted some
temporary solution for that (running at each dpkg invocation).

Ok, so for now (Whonix 11 templates) I see three options:

1. We find a way to implement it properly.
2. We stay with current solution accepting some inaccuracy in that
   reporting
3. We disable all the "updates pending" notification for Debian-based
   templates (not only Whonix ones).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

Does apt-get have an 'all upgrades installed' hook?:
https://lists.debian.org/debian-user/2015/08/msg00198.html

---

Yes, a polling mechanism would be rather hacky.

This would require running all the updates through PackageKit, right?

I'll ask.

---

Ok, so for now (Whonix 11 templates) I see three options:
We find a way to implement it properly.

For a later release, yes.

We stay with current solution accepting some inaccuracy in that reporting

Alright.

We disable all the "updates pending" notification for Debian-based templates (not only Whonix ones).

If the indicator is broken, yes.

[adrelanos]

Does apt-get have an 'all upgrades installed' hook?:
https://lists.debian.org/debian-user/2015/08/msg00198.html

---

Yes, a polling mechanism would be rather hacky.

This would require running all the updates through PackageKit, right?

I'll ask.

---

Ok, so for now (Whonix 11 templates) I see three options:
We find a way to implement it properly.

For a later release, yes.

We stay with current solution accepting some inaccuracy in that reporting

Alright.

We disable all the "updates pending" notification for Debian-based templates (not only Whonix ones).

If the indicator is broken, yes.

[adrelanos]

Debian apt-get 'all upgrades installed' hook?:
http://lists.freedesktop.org/archives/packagekit/2015-August/026378.html

[adrelanos]

Debian apt-get 'all upgrades installed' hook?:
http://lists.freedesktop.org/archives/packagekit/2015-August/026378.html

[nrgaway]

On 6 August 2015 at 11:11, Patrick Schleizer notifications@github.com
wrote:

This caused a whonix-gw-experimental VM to keep apt-get locked for several
minutes.

ps aux | grep apt
root 2939 0.0 0.1 18936 688 ? SLs 14:18 0:00 /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2940 0.0 0.0 4192 500 ? S 14:18 0:00 sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2941 0.0 0.7 24132 4496 ? S 14:18 0:00 /usr/bin/apt-get.anondist-orig -q update
root 2958 0.0 0.6 23600 3404 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2959 0.0 0.5 23600 3304 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2960 0.0 0.6 23600 3416 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2961 0.4 0.6 23600 3636 ? S 14:18 0:01 /usr/lib/apt/methods/http
root 2962 0.0 0.6 23600 3780 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 3029 0.0 0.6 23564 3388 ? S 14:18 0:00 /usr/lib/apt/methods/gpgv
root 3036 0.0 0.6 23552 3448 ? S 14:18 0:00 /usr/lib/apt/methods/copy
root 3038 0.0 0.5 23552 3336 ? S 14:18 0:00 /usr/lib/apt/methods/gzip
root 3045 0.0 0.9 27160 5380 ? S 14:18 0:00 /usr/lib/apt/methods/bzip2
root 3117 0.7 0.6 23552 3596 ? S 14:21 0:00 /usr/lib/apt/methods/xz
user 3145 0.0 0.3 7844 1992 pts/0 S+ 14:22 0:00 grep --color=auto apt

This happens once for both Fedora and Debian. A timer is set to do an
apt-get update 5 minutes after the VM starts.

[nrgaway]

On 6 August 2015 at 11:11, Patrick Schleizer notifications@github.com
wrote:

This caused a whonix-gw-experimental VM to keep apt-get locked for several
minutes.

ps aux | grep apt
root 2939 0.0 0.1 18936 688 ? SLs 14:18 0:00 /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2940 0.0 0.0 4192 500 ? S 14:18 0:00 sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2941 0.0 0.7 24132 4496 ? S 14:18 0:00 /usr/bin/apt-get.anondist-orig -q update
root 2958 0.0 0.6 23600 3404 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2959 0.0 0.5 23600 3304 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2960 0.0 0.6 23600 3416 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2961 0.4 0.6 23600 3636 ? S 14:18 0:01 /usr/lib/apt/methods/http
root 2962 0.0 0.6 23600 3780 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 3029 0.0 0.6 23564 3388 ? S 14:18 0:00 /usr/lib/apt/methods/gpgv
root 3036 0.0 0.6 23552 3448 ? S 14:18 0:00 /usr/lib/apt/methods/copy
root 3038 0.0 0.5 23552 3336 ? S 14:18 0:00 /usr/lib/apt/methods/gzip
root 3045 0.0 0.9 27160 5380 ? S 14:18 0:00 /usr/lib/apt/methods/bzip2
root 3117 0.7 0.6 23552 3596 ? S 14:21 0:00 /usr/lib/apt/methods/xz
user 3145 0.0 0.3 7844 1992 pts/0 S+ 14:22 0:00 grep --color=auto apt

This happens once for both Fedora and Debian. A timer is set to do an
apt-get update 5 minutes after the VM starts.

[nrgaway]

On 6 August 2015 at 17:59, Patrick Schleizer notifications@github.com
wrote:

packagekit: Had a quick lock at the API. Most likely could be used to
implement this. I think that, because Apper, which is based on packagekit
can find out how many upgrades are available. However, I guess a packagekit
based solution would require non-negligible development effort. There is
python-packagekit, installable from Debian repository if anyone wants to
look into it.

The latest version of packagekit ripped out the update code it seems. They
use an apt-get hook that does auto updates; which is similar to what we do;
some type of timer to check for updates.

The problem we have is that TemplateVM is powered off most of the time, and
a apt-get update needs to happen at some point. In you message above you
were experiencing the update taking place about 5 minutes in. packagekit
has a command line utility you can checkout called pkcon.

The current notify hook may run more than once, but its very quick and not
noticeable.

There is a way to call packagekit that will first to an update. I have not
found out how to do it with the latest version yet.

[nrgaway]

On 6 August 2015 at 17:59, Patrick Schleizer notifications@github.com
wrote:

packagekit: Had a quick lock at the API. Most likely could be used to
implement this. I think that, because Apper, which is based on packagekit
can find out how many upgrades are available. However, I guess a packagekit
based solution would require non-negligible development effort. There is
python-packagekit, installable from Debian repository if anyone wants to
look into it.

The latest version of packagekit ripped out the update code it seems. They
use an apt-get hook that does auto updates; which is similar to what we do;
some type of timer to check for updates.

The problem we have is that TemplateVM is powered off most of the time, and
a apt-get update needs to happen at some point. In you message above you
were experiencing the update taking place about 5 minutes in. packagekit
has a command line utility you can checkout called pkcon.

The current notify hook may run more than once, but its very quick and not
noticeable.

There is a way to call packagekit that will first to an update. I have not
found out how to do it with the latest version yet.

[nrgaway]

...

We disable all the "updates pending" notification for Debian-based
templates (not only Whonix ones).

If the indicator is broken, yes.

The indicator works fine here. I find it useful to know when to update.
When I need to update, I start the TemplateVM and wait for about 7
minutes. Since the timer runs 5 minutes in to update the package index, I
run gpk and they updates show; then I update. The status indicator also
disappears from qubes-manager.

[nrgaway]

...

We disable all the "updates pending" notification for Debian-based
templates (not only Whonix ones).

If the indicator is broken, yes.

The indicator works fine here. I find it useful to know when to update.
When I need to update, I start the TemplateVM and wait for about 7
minutes. Since the timer runs 5 minutes in to update the package index, I
run gpk and they updates show; then I update. The status indicator also
disappears from qubes-manager.

[adrelanos]

The whole packagekit thread:
http://lists.freedesktop.org/archives/packagekit/2015-August/026378.html

A relevant answer, that I am copying in here for your convenience:
http://lists.freedesktop.org/archives/packagekit/2015-August/026381.html

Daniel Nicoletti:

Packagekit aptcc backend does exactly what you want,
but I'm thinking on changing that in future to show only
upgrade instead of dist-upgrade (which can easily lead
to broken instalations).

if gettimeSinceLastUpdate() returns a short value and
getUpdates() is called you will have a list of dist-upgraded
packages.

In future I'd emit the additional packages that dist-upgrade
emits as blocked, so the user knows there are more upgrades
but those need manual selection.

Best.

[adrelanos]

The whole packagekit thread:
http://lists.freedesktop.org/archives/packagekit/2015-August/026378.html

A relevant answer, that I am copying in here for your convenience:
http://lists.freedesktop.org/archives/packagekit/2015-August/026381.html

Daniel Nicoletti:

Packagekit aptcc backend does exactly what you want,
but I'm thinking on changing that in future to show only
upgrade instead of dist-upgrade (which can easily lead
to broken instalations).

if gettimeSinceLastUpdate() returns a short value and
getUpdates() is called you will have a list of dist-upgraded
packages.

In future I'd emit the additional packages that dist-upgrade
emits as blocked, so the user knows there are more upgrades
but those need manual selection.

Best.

[nrgaway]

BTW, This is the apt code they use to inform packagekit that updates are complete...

// THIS FILE IS USED TO INFORM PACKAGEKIT
// THAT THE UPDATE-INFO MIGHT HAVE CHANGED

// Whenever dpkg is called we might have different updates
// i.e. if an user removes a package that had an update
DPkg::Post-Invoke {
"/usr/bin/test -e /usr/share/dbus-1/system-services/org.freedesktop.PackageKit.service && /usr/bin/test -S /var/run/dbus/system_bus_socket && /usr/bin/gdbus call --system --dest org.freedesktop.PackageKit --object-path /org/freedesktop/PackageKit --timeout 4 --method org.freedesktop.PackageKit.StateHasChanged cache-update > /dev/null; /bin/echo > /dev/null";
};

// When Apt's cache is updated (i.e. apt-cache update)
APT::Update::Post-Invoke-Success {
"/usr/bin/test -e /usr/share/dbus-1/system-services/org.freedesktop.PackageKit.service && /usr/bin/test -S /var/run/dbus/system_bus_socket && /usr/bin/gdbus call --system --dest org.freedesktop.PackageKit --object-path /org/freedesktop/PackageKit --timeout 4 --method org.freedesktop.PackageKit.StateHasChanged cache-update > /dev/null; /bin/echo > /dev/null";
};

[nrgaway]

BTW, This is the apt code they use to inform packagekit that updates are complete...

// THIS FILE IS USED TO INFORM PACKAGEKIT
// THAT THE UPDATE-INFO MIGHT HAVE CHANGED

// Whenever dpkg is called we might have different updates
// i.e. if an user removes a package that had an update
DPkg::Post-Invoke {
"/usr/bin/test -e /usr/share/dbus-1/system-services/org.freedesktop.PackageKit.service && /usr/bin/test -S /var/run/dbus/system_bus_socket && /usr/bin/gdbus call --system --dest org.freedesktop.PackageKit --object-path /org/freedesktop/PackageKit --timeout 4 --method org.freedesktop.PackageKit.StateHasChanged cache-update > /dev/null; /bin/echo > /dev/null";
};

// When Apt's cache is updated (i.e. apt-cache update)
APT::Update::Post-Invoke-Success {
"/usr/bin/test -e /usr/share/dbus-1/system-services/org.freedesktop.PackageKit.service && /usr/bin/test -S /var/run/dbus/system_bus_socket && /usr/bin/gdbus call --system --dest org.freedesktop.PackageKit --object-path /org/freedesktop/PackageKit --timeout 4 --method org.freedesktop.PackageKit.StateHasChanged cache-update > /dev/null; /bin/echo > /dev/null";
};

[adrelanos]

nrgaway:

On 6 August 2015 at 11:11, Patrick Schleizer notifications@github.com
wrote:

This caused a whonix-gw-experimental VM to keep apt-get locked for several
minutes.

ps aux | grep apt
root 2939 0.0 0.1 18936 688 ? SLs 14:18 0:00 /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2940 0.0 0.0 4192 500 ? S 14:18 0:00 sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2941 0.0 0.7 24132 4496 ? S 14:18 0:00 /usr/bin/apt-get.anondist-orig -q update
root 2958 0.0 0.6 23600 3404 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2959 0.0 0.5 23600 3304 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2960 0.0 0.6 23600 3416 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2961 0.4 0.6 23600 3636 ? S 14:18 0:01 /usr/lib/apt/methods/http
root 2962 0.0 0.6 23600 3780 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 3029 0.0 0.6 23564 3388 ? S 14:18 0:00 /usr/lib/apt/methods/gpgv
root 3036 0.0 0.6 23552 3448 ? S 14:18 0:00 /usr/lib/apt/methods/copy
root 3038 0.0 0.5 23552 3336 ? S 14:18 0:00 /usr/lib/apt/methods/gzip
root 3045 0.0 0.9 27160 5380 ? S 14:18 0:00 /usr/lib/apt/methods/bzip2
root 3117 0.7 0.6 23552 3596 ? S 14:21 0:00 /usr/lib/apt/methods/xz
user 3145 0.0 0.3 7844 1992 pts/0 S+ 14:22 0:00 grep --color=auto apt

This happens once for both Fedora and Debian. A timer is set to do an
apt-get update 5 minutes after the VM starts.

But isn't that a usability issue? Because there is no feedback to the
user that it's currently running 'apt-get update', checking for
upgrades. The locked apt-get that I noticed is a usability issue. This
has been made visible due to slow networking. So for users with slow
connections [villages, GPRS, developing countries...] - does Qubes care
about this? And an issue for people doing torified updates (Whonix,
torification of all outgoing traffic).

On the other hand, if you dispatched a passive popup every time you are
running 'apt-get update', checking for upgrades, that would be too much
also. I don't know how useful a status indicator in Qubes VM Manager
(checking for upgrades) would be, because it's not visible the moment
the user wonders about the locked apt-get.

By the way, this is why in Whonix, whonixcheck does the automated
running of apt-get update and checking for available upgrades loudly by
opening a progress bar.

[adrelanos]

nrgaway:

On 6 August 2015 at 11:11, Patrick Schleizer notifications@github.com
wrote:

This caused a whonix-gw-experimental VM to keep apt-get locked for several
minutes.

ps aux | grep apt
root 2939 0.0 0.1 18936 688 ? SLs 14:18 0:00 /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2940 0.0 0.0 4192 500 ? S 14:18 0:00 sh -c if [ -e /etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ] && echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2941 0.0 0.7 24132 4496 ? S 14:18 0:00 /usr/bin/apt-get.anondist-orig -q update
root 2958 0.0 0.6 23600 3404 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2959 0.0 0.5 23600 3304 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2960 0.0 0.6 23600 3416 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2961 0.4 0.6 23600 3636 ? S 14:18 0:01 /usr/lib/apt/methods/http
root 2962 0.0 0.6 23600 3780 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 3029 0.0 0.6 23564 3388 ? S 14:18 0:00 /usr/lib/apt/methods/gpgv
root 3036 0.0 0.6 23552 3448 ? S 14:18 0:00 /usr/lib/apt/methods/copy
root 3038 0.0 0.5 23552 3336 ? S 14:18 0:00 /usr/lib/apt/methods/gzip
root 3045 0.0 0.9 27160 5380 ? S 14:18 0:00 /usr/lib/apt/methods/bzip2
root 3117 0.7 0.6 23552 3596 ? S 14:21 0:00 /usr/lib/apt/methods/xz
user 3145 0.0 0.3 7844 1992 pts/0 S+ 14:22 0:00 grep --color=auto apt

This happens once for both Fedora and Debian. A timer is set to do an
apt-get update 5 minutes after the VM starts.

But isn't that a usability issue? Because there is no feedback to the
user that it's currently running 'apt-get update', checking for
upgrades. The locked apt-get that I noticed is a usability issue. This
has been made visible due to slow networking. So for users with slow
connections [villages, GPRS, developing countries...] - does Qubes care
about this? And an issue for people doing torified updates (Whonix,
torification of all outgoing traffic).

On the other hand, if you dispatched a passive popup every time you are
running 'apt-get update', checking for upgrades, that would be too much
also. I don't know how useful a status indicator in Qubes VM Manager
(checking for upgrades) would be, because it's not visible the moment
the user wonders about the locked apt-get.

By the way, this is why in Whonix, whonixcheck does the automated
running of apt-get update and checking for available upgrades loudly by
opening a progress bar.

[nrgaway]

On 7 August 2015 at 13:44, Patrick Schleizer notifications@github.com
wrote:

nrgaway:

On 6 August 2015 at 11:11, Patrick Schleizer notifications@github.com
wrote:

This caused a whonix-gw-experimental VM to keep apt-get locked for
several
minutes.

ps aux | grep apt
root 2939 0.0 0.1 18936 688 ? SLs 14:18 0:00
/usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c if [ -e
/etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ]
&& echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade
| awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2940 0.0 0.0 4192 500 ? S 14:18 0:00 sh -c if [ -e
/etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ]
&& echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade
| awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2941 0.0 0.7 24132 4496 ? S 14:18 0:00
/usr/bin/apt-get.anondist-orig -q update
root 2958 0.0 0.6 23600 3404 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2959 0.0 0.5 23600 3304 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2960 0.0 0.6 23600 3416 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2961 0.4 0.6 23600 3636 ? S 14:18 0:01 /usr/lib/apt/methods/http
root 2962 0.0 0.6 23600 3780 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 3029 0.0 0.6 23564 3388 ? S 14:18 0:00 /usr/lib/apt/methods/gpgv
root 3036 0.0 0.6 23552 3448 ? S 14:18 0:00 /usr/lib/apt/methods/copy
root 3038 0.0 0.5 23552 3336 ? S 14:18 0:00 /usr/lib/apt/methods/gzip
root 3045 0.0 0.9 27160 5380 ? S 14:18 0:00 /usr/lib/apt/methods/bzip2
root 3117 0.7 0.6 23552 3596 ? S 14:21 0:00 /usr/lib/apt/methods/xz
user 3145 0.0 0.3 7844 1992 pts/0 S+ 14:22 0:00 grep --color=auto apt

This happens once for both Fedora and Debian. A timer is set to do an
apt-get update 5 minutes after the VM starts.

But isn't that a usability issue? Because there is no feedback to the
user that it's currently running 'apt-get update', checking for
upgrades. The locked apt-get that I noticed is a usability issue. This
has been made visible due to slow networking. So for users with slow
connections [villages, GPRS, developing countries...] - does Qubes care
about this? And an issue for people doing torified updates (Whonix,
torification of all outgoing traffic).

Yes, I assume whoever implemented that feature to begin with did it to
benefit the user.

Packagekit used to check for updates when it was run; still does in Fedora
I believe. When the 'Update VM' button is pressed it would be most
desirable for Gnome PackageKit to obtain the index update and display a
status bar of its progress.

GPK User site say there is a way to start update though the system menu. I
figure maybe installing an HVM to see if that is still working with latest
version.

On the other hand, if you dispatched a passive popup every time you are
running 'apt-get update', checking for upgrades, that would be too much
also. I don't know how useful a status indicator in Qubes VM Manager
(checking for upgrades) would be, because it's not visible the moment
the user wonders about the locked apt-get.

By the way, this is why in Whonix, whonixcheck does the automated
running of apt-get update and checking for available upgrades loudly by
opening a progress bar.

[nrgaway]

On 7 August 2015 at 13:44, Patrick Schleizer notifications@github.com
wrote:

nrgaway:

On 6 August 2015 at 11:11, Patrick Schleizer notifications@github.com
wrote:

This caused a whonix-gw-experimental VM to keep apt-get locked for
several
minutes.

ps aux | grep apt
root 2939 0.0 0.1 18936 688 ? SLs 14:18 0:00
/usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c if [ -e
/etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ]
&& echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade
| awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2940 0.0 0.0 4192 500 ? S 14:18 0:00 sh -c if [ -e
/etc/system-release ]; then yum -q check-update >/dev/null; [ $? -eq 100 ]
&& echo 1 || echo 0; else apt-get -q update > /dev/null; apt-get -s upgrade
| awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1; fi
root 2941 0.0 0.7 24132 4496 ? S 14:18 0:00
/usr/bin/apt-get.anondist-orig -q update
root 2958 0.0 0.6 23600 3404 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2959 0.0 0.5 23600 3304 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2960 0.0 0.6 23600 3416 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 2961 0.4 0.6 23600 3636 ? S 14:18 0:01 /usr/lib/apt/methods/http
root 2962 0.0 0.6 23600 3780 ? S 14:18 0:00 /usr/lib/apt/methods/http
root 3029 0.0 0.6 23564 3388 ? S 14:18 0:00 /usr/lib/apt/methods/gpgv
root 3036 0.0 0.6 23552 3448 ? S 14:18 0:00 /usr/lib/apt/methods/copy
root 3038 0.0 0.5 23552 3336 ? S 14:18 0:00 /usr/lib/apt/methods/gzip
root 3045 0.0 0.9 27160 5380 ? S 14:18 0:00 /usr/lib/apt/methods/bzip2
root 3117 0.7 0.6 23552 3596 ? S 14:21 0:00 /usr/lib/apt/methods/xz
user 3145 0.0 0.3 7844 1992 pts/0 S+ 14:22 0:00 grep --color=auto apt

This happens once for both Fedora and Debian. A timer is set to do an
apt-get update 5 minutes after the VM starts.

But isn't that a usability issue? Because there is no feedback to the
user that it's currently running 'apt-get update', checking for
upgrades. The locked apt-get that I noticed is a usability issue. This
has been made visible due to slow networking. So for users with slow
connections [villages, GPRS, developing countries...] - does Qubes care
about this? And an issue for people doing torified updates (Whonix,
torification of all outgoing traffic).

Yes, I assume whoever implemented that feature to begin with did it to
benefit the user.

Packagekit used to check for updates when it was run; still does in Fedora
I believe. When the 'Update VM' button is pressed it would be most
desirable for Gnome PackageKit to obtain the index update and display a
status bar of its progress.

GPK User site say there is a way to start update though the system menu. I
figure maybe installing an HVM to see if that is still working with latest
version.

On the other hand, if you dispatched a passive popup every time you are
running 'apt-get update', checking for upgrades, that would be too much
also. I don't know how useful a status indicator in Qubes VM Manager
(checking for upgrades) would be, because it's not visible the moment
the user wonders about the locked apt-get.

By the way, this is why in Whonix, whonixcheck does the automated
running of apt-get update and checking for available upgrades loudly by
opening a progress bar.

[marmarek]

On Fri, Aug 07, 2015 at 10:44:17AM -0700, Patrick Schleizer wrote:

But isn't that a usability issue? Because there is no feedback to the
user that it's currently running 'apt-get update', checking for
upgrades. The locked apt-get that I noticed is a usability issue. This
has been made visible due to slow networking. So for users with slow
connections [villages, GPRS, developing countries...] - does Qubes care
about this? And an issue for people doing torified updates (Whonix,
torification of all outgoing traffic).

This is also exactly what PackageKit is going to use - periodic call to
apt-get update, which actually already exists in the apt package
itself. Take a look at /etc/cron.daily/apt.

So perhaps our script could be reduced to just call apt-get -s upgrade?
Or if we could have PackageKit hook/callback for both actions ("new
updates available" and "all updates installed"), we could get rid of
this (periodically checking for updates), as well custom apt/dpkg hook
and rely on PackageKit to report all of that.

But still didn't figured out how to get it from PackageKit (see my
response on packagekit ML).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Fri, Aug 07, 2015 at 10:44:17AM -0700, Patrick Schleizer wrote:

But isn't that a usability issue? Because there is no feedback to the
user that it's currently running 'apt-get update', checking for
upgrades. The locked apt-get that I noticed is a usability issue. This
has been made visible due to slow networking. So for users with slow
connections [villages, GPRS, developing countries...] - does Qubes care
about this? And an issue for people doing torified updates (Whonix,
torification of all outgoing traffic).

This is also exactly what PackageKit is going to use - periodic call to
apt-get update, which actually already exists in the apt package
itself. Take a look at /etc/cron.daily/apt.

So perhaps our script could be reduced to just call apt-get -s upgrade?
Or if we could have PackageKit hook/callback for both actions ("new
updates available" and "all updates installed"), we could get rid of
this (periodically checking for updates), as well custom apt/dpkg hook
and rely on PackageKit to report all of that.

But still didn't figured out how to get it from PackageKit (see my
response on packagekit ML).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

Does apt-get have an 'all upgrades installed' hook?:
https://lists.debian.org/debian-user/2015/08/msg00198.html

No answer.

But still didn't figured out how to get it from PackageKit (see my response on packagekit ML).
http://lists.freedesktop.org/archives/packagekit/2015-August/026383.html

No answer.

[adrelanos]

Does apt-get have an 'all upgrades installed' hook?:
https://lists.debian.org/debian-user/2015/08/msg00198.html

No answer.

But still didn't figured out how to get it from PackageKit (see my response on packagekit ML).
http://lists.freedesktop.org/archives/packagekit/2015-August/026383.html

No answer.

[adrelanos]

Talking about /etc/apt/apt.conf.d/00notify-hook in this comment.
(snapshot: https://github.com/QubesOS/qubes-core-agent-linux/blob/15f1df49474368510180c8b232a69c2a5dce9533/network/00notify-hook)

Problem:
Each time some arbitrary package is installed using dpkg or apt-get, the update notification in Qubes VM Manager is cleared. No matter if there are still updates pending. (Can happen even after the user running apt-get dist-upgrade in case of package manager issues.)

Approaches:
I don't think there is a hook that can be dispatched at the end/after of apt-get. It's been discussed in user forums. The answers are:

• a) shell alias's
  • probably not a good solution for Qubes (shell alias's are not always load)
• b) renaming (dpkg-divert / config-package-dev etc.) /usr/bin/apt-get with a hook script
  • not great, Whonix already does this to inject uwt/socks proxy settings/stream isolation/torsocks. And two dpkg diverts would conflict with each other]. (apt-get really does not support socks proxy settings [Acquire::socks::proxy is a myth. socks is not mentioned in apt-get's source code)
• c) /usr/local/bin/apt-get hook script
  • not great, because it's in /rw in Qubes, therefore /usr/local/bin/apt-get could not be managed by Qubes?

a), b) and c) are all not great. But in the absence of a hook.. A periodic check, let's say checking every 5 minutes if all packages have been installed running essentially apt-get -s upgrade | awk "/^Inst/{ print $2 }" would make users wonder why the updates required flag is not instantly cleared from Qubes VM Manager after they successfully finished running apt-get dist-upgrade. But running apt-get -s upgrade | awk "/^Inst/{ print $2 }" every 60 seconds or less would result in a huge waste of recourses (accumulates with every VM).

---

I was wrong. DPkg::Post-Invoke just runs once when run by apt-get after package install/removal.

DPkg::Post-Invoke-Success is a myth. Does not work and cannot be found in apt-gets source code.

---

Suggested solution:

/etc/apt/apt.conf.d/00notify-hook:

DPkg::Post-Invoke {"/usr/lib/qubes/updates-notify-clear-maybe";};

/usr/lib/qubes/updates-notify-clear-maybe:

#!/bin/bash
updates_available="$(/usr/lib/qubes/upgrades-installed-check)"
if [ "$updates_available" = "1" ]; then
   /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c 'echo 0'
fi
exit 0

/usr/lib/qubes/upgrades-installed-check

#!/bin/bash

if [ -e /etc/system-release ]; then
   ## Fedora
   yum -q check-update >/dev/null;
   [ $? -eq 100 ] && echo 1 || echo 0;
else
   ## Debian
   set -e
   set -o pipefail
   apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1;
fi

It's partially tested. What do you think? Can this all be added to qubes-core-agent? Any modification or file name rename requests?

[adrelanos]

Talking about /etc/apt/apt.conf.d/00notify-hook in this comment.
(snapshot: https://github.com/QubesOS/qubes-core-agent-linux/blob/15f1df49474368510180c8b232a69c2a5dce9533/network/00notify-hook)

Problem:
Each time some arbitrary package is installed using dpkg or apt-get, the update notification in Qubes VM Manager is cleared. No matter if there are still updates pending. (Can happen even after the user running apt-get dist-upgrade in case of package manager issues.)

Approaches:
I don't think there is a hook that can be dispatched at the end/after of apt-get. It's been discussed in user forums. The answers are:

• a) shell alias's
  • probably not a good solution for Qubes (shell alias's are not always load)
• b) renaming (dpkg-divert / config-package-dev etc.) /usr/bin/apt-get with a hook script
  • not great, Whonix already does this to inject uwt/socks proxy settings/stream isolation/torsocks. And two dpkg diverts would conflict with each other]. (apt-get really does not support socks proxy settings [Acquire::socks::proxy is a myth. socks is not mentioned in apt-get's source code)
• c) /usr/local/bin/apt-get hook script
  • not great, because it's in /rw in Qubes, therefore /usr/local/bin/apt-get could not be managed by Qubes?

a), b) and c) are all not great. But in the absence of a hook.. A periodic check, let's say checking every 5 minutes if all packages have been installed running essentially apt-get -s upgrade | awk "/^Inst/{ print $2 }" would make users wonder why the updates required flag is not instantly cleared from Qubes VM Manager after they successfully finished running apt-get dist-upgrade. But running apt-get -s upgrade | awk "/^Inst/{ print $2 }" every 60 seconds or less would result in a huge waste of recourses (accumulates with every VM).

---

I was wrong. DPkg::Post-Invoke just runs once when run by apt-get after package install/removal.

DPkg::Post-Invoke-Success is a myth. Does not work and cannot be found in apt-gets source code.

---

Suggested solution:

/etc/apt/apt.conf.d/00notify-hook:

DPkg::Post-Invoke {"/usr/lib/qubes/updates-notify-clear-maybe";};

/usr/lib/qubes/updates-notify-clear-maybe:

#!/bin/bash
updates_available="$(/usr/lib/qubes/upgrades-installed-check)"
if [ "$updates_available" = "1" ]; then
   /usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c 'echo 0'
fi
exit 0

/usr/lib/qubes/upgrades-installed-check

#!/bin/bash

if [ -e /etc/system-release ]; then
   ## Fedora
   yum -q check-update >/dev/null;
   [ $? -eq 100 ] && echo 1 || echo 0;
else
   ## Debian
   set -e
   set -o pipefail
   apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1;
fi

It's partially tested. What do you think? Can this all be added to qubes-core-agent? Any modification or file name rename requests?

[marmarek]

On Wed, Oct 21, 2015 at 03:40:53PM -0700, Patrick Schleizer wrote:

a), b) and c) are all not great. But in the absence of a hook.. A periodic check, let's say checking every 5 minutes if all packages have been installed running essentially apt-get -s upgrade | awk "/^Inst/{ print $2 }" would make users wonder why the updates required flag is not instantly cleared from Qubes VM Manager after they successfully finished running apt-get dist-upgrade. But running apt-get -s upgrade | awk "/^Inst/{ print $2 }" every 60 seconds or less would result in a huge waste of recourses (accumulates with every VM).

Additionally it would not work with IMO typical workflow:
sudo apt-get update
sudo apt-get dist-upgrade
sudo poweroff # very little or no time after previous command

---

I was wrong. DPkg::Post-Invoke just runs once when run by apt-get after package install/removal.

Great!

/usr/lib/qubes/updates-notify-clear-maybe:

#!/bin/bash
updates_available="$(/usr/lib/qubes/upgrades-installed-check)"

updates_installed=

if [ "$updates_available" = "1" ]; then
/usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c 'echo 0'
fi
exit 0

Should check in both directions: send non-zero if some updates are still
not installed. This way it will correctly report when for example user
in VM calls just apt-get update to initially mark template as needing
update.

/usr/lib/qubes/upgrades-installed-check

#!/bin/bash

if [ -e /etc/system-release ]; then
   ## Fedora
   yum -q check-update >/dev/null;
   [ $? -eq 100 ] && echo 1 || echo 0;
else
   ## Debian
   set -e
   set -o pipefail
   apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1;
fi

I'm thinking about some general approach here: should we have generic
scripts which handle all the distributions (your example), or have
separate one for each supported distribution and include them in
appropriate packages - so Debian package will carry Debian-specific script
here, while Fedora - it's own, in the same place.
I'm slightly for the current approach, but just a thought.

It's partially tested. What do you think? Can this all be added to qubes-core-agent? Any modification or file name rename requests?

It all generally looks good. One more issue is what happens when user
starts the template and call just apt-get dist-upgrade, without
apt-get update first. This will clear "updates pending" marker, while
actually not install any update. Any idea how to solve this?

IMHO we can just ignore the problem ("updates pending" flag will
reappear shortly, when some VM will report that). Additionally when
we'll get working GUI updates, it should not be a problem anymore,
because that tool will take care of calling apt-get update.

Related:
#1249

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Wed, Oct 21, 2015 at 03:40:53PM -0700, Patrick Schleizer wrote:

a), b) and c) are all not great. But in the absence of a hook.. A periodic check, let's say checking every 5 minutes if all packages have been installed running essentially apt-get -s upgrade | awk "/^Inst/{ print $2 }" would make users wonder why the updates required flag is not instantly cleared from Qubes VM Manager after they successfully finished running apt-get dist-upgrade. But running apt-get -s upgrade | awk "/^Inst/{ print $2 }" every 60 seconds or less would result in a huge waste of recourses (accumulates with every VM).

Additionally it would not work with IMO typical workflow:
sudo apt-get update
sudo apt-get dist-upgrade
sudo poweroff # very little or no time after previous command

---

I was wrong. DPkg::Post-Invoke just runs once when run by apt-get after package install/removal.

Great!

/usr/lib/qubes/updates-notify-clear-maybe:

#!/bin/bash
updates_available="$(/usr/lib/qubes/upgrades-installed-check)"

updates_installed=

if [ "$updates_available" = "1" ]; then
/usr/lib/qubes/qrexec-client-vm dom0 qubes.NotifyUpdates /bin/sh -c 'echo 0'
fi
exit 0

Should check in both directions: send non-zero if some updates are still
not installed. This way it will correctly report when for example user
in VM calls just apt-get update to initially mark template as needing
update.

/usr/lib/qubes/upgrades-installed-check

#!/bin/bash

if [ -e /etc/system-release ]; then
   ## Fedora
   yum -q check-update >/dev/null;
   [ $? -eq 100 ] && echo 1 || echo 0;
else
   ## Debian
   set -e
   set -o pipefail
   apt-get -s upgrade | awk "/^Inst/{ print $2 }" | [ $(wc -L) -eq 0 ] && echo 0 || echo 1;
fi

I'm thinking about some general approach here: should we have generic
scripts which handle all the distributions (your example), or have
separate one for each supported distribution and include them in
appropriate packages - so Debian package will carry Debian-specific script
here, while Fedora - it's own, in the same place.
I'm slightly for the current approach, but just a thought.

It's partially tested. What do you think? Can this all be added to qubes-core-agent? Any modification or file name rename requests?

It all generally looks good. One more issue is what happens when user
starts the template and call just apt-get dist-upgrade, without
apt-get update first. This will clear "updates pending" marker, while
actually not install any update. Any idea how to solve this?

IMHO we can just ignore the problem ("updates pending" flag will
reappear shortly, when some VM will report that). Additionally when
we'll get working GUI updates, it should not be a problem anymore,
because that tool will take care of calling apt-get update.

Related:
#1249

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

FYI:

• /var/lib/apt/lists/ - small, lists only
• /var/cache/apt/ - huge, contains cached pages

---

Marek Marczykowski-Górecki:

I'm thinking about some general approach here: should we have generic
scripts which handle all the distributions (your example), or have
separate one for each supported distribution and include them in
appropriate packages - so Debian package will carry Debian-specific script
here, while Fedora - it's own, in the same place.
I'm slightly for the current approach, but just a thought.

I'm also slightly for the current approach on just using one script. It
makes ports easier. Someone trying to re-implement apt-get hook for lets
say yum can reuse the same path to the hook with less thinking. If the
script would get really huge however it might make sense to split it.
But for now its easier to say "just add a hook and extend that script".

It all generally looks good. One more issue is what happens when user
starts the template and call just apt-get dist-upgrade, without
apt-get update first. This will clear "updates pending" marker, while
actually not install any update. Any idea how to solve this?

The problem is, that in TemplateBased VMs, /var/lib/apt is not
persistent. Therefore anything inside there is "forgotten" after the
restart of a TemplateBasedVM.

Three ideas...

---

a)

How (in)sane does it sound to,

• on boot, maybe mirror /var/lib/apt to /rw/qubes-internal/var/lib/apt?
• Then /usr/lib/qubes/upgrades-installed-check's apt-get -s upgrade
  could be pointed to /rw/qubes-internal/var/lib/apt using apt's
  directory options (-o Dir::State etc.).

But it might be difficult.

• When on another boot /var/lib/apt in the root image is newer than
  /rw/qubes-internal/var/lib/apt, then this means, that 'apt-get update'
  was run in the TemplateVM. Then /rw/qubes-internal/var/lib/apt should
  be discarded and replaced by /var/lib/apt.

---

b)

Alternatively I am wondering if making whole /var/lib/apt would be sane?

---

c)

• The apt-get update run by Qubes update check could be pointed to
  /rw/qubes-internal/pkg/apt using apt's directory options.
• /usr/lib/qubes/upgrades-installed-check's apt-get -s upgrade would
  also be pointed there.

I haven't thought though yet if any desync issues / corner cases could
lead to an inconsistency between /var/cache/apt and
/rw/qubes-internal/pkg/apt. I.e. user running apt-get claiming
something other than Qubes update check.

---

But let's not get lost into this. I think we should create a separate
issue for this? Because the issue I recently described in my last post
is a legitimate one and the fix suggested is an improvement for the problem.

[adrelanos]

FYI:

• /var/lib/apt/lists/ - small, lists only
• /var/cache/apt/ - huge, contains cached pages

---

Marek Marczykowski-Górecki:

I'm thinking about some general approach here: should we have generic
scripts which handle all the distributions (your example), or have
separate one for each supported distribution and include them in
appropriate packages - so Debian package will carry Debian-specific script
here, while Fedora - it's own, in the same place.
I'm slightly for the current approach, but just a thought.

I'm also slightly for the current approach on just using one script. It
makes ports easier. Someone trying to re-implement apt-get hook for lets
say yum can reuse the same path to the hook with less thinking. If the
script would get really huge however it might make sense to split it.
But for now its easier to say "just add a hook and extend that script".

It all generally looks good. One more issue is what happens when user
starts the template and call just apt-get dist-upgrade, without
apt-get update first. This will clear "updates pending" marker, while
actually not install any update. Any idea how to solve this?

The problem is, that in TemplateBased VMs, /var/lib/apt is not
persistent. Therefore anything inside there is "forgotten" after the
restart of a TemplateBasedVM.

Three ideas...

---

a)

How (in)sane does it sound to,

• on boot, maybe mirror /var/lib/apt to /rw/qubes-internal/var/lib/apt?
• Then /usr/lib/qubes/upgrades-installed-check's apt-get -s upgrade
  could be pointed to /rw/qubes-internal/var/lib/apt using apt's
  directory options (-o Dir::State etc.).

But it might be difficult.

• When on another boot /var/lib/apt in the root image is newer than
  /rw/qubes-internal/var/lib/apt, then this means, that 'apt-get update'
  was run in the TemplateVM. Then /rw/qubes-internal/var/lib/apt should
  be discarded and replaced by /var/lib/apt.

---

b)

Alternatively I am wondering if making whole /var/lib/apt would be sane?

---

c)

• The apt-get update run by Qubes update check could be pointed to
  /rw/qubes-internal/pkg/apt using apt's directory options.
• /usr/lib/qubes/upgrades-installed-check's apt-get -s upgrade would
  also be pointed there.

I haven't thought though yet if any desync issues / corner cases could
lead to an inconsistency between /var/cache/apt and
/rw/qubes-internal/pkg/apt. I.e. user running apt-get claiming
something other than Qubes update check.

---

But let's not get lost into this. I think we should create a separate
issue for this? Because the issue I recently described in my last post
is a legitimate one and the fix suggested is an improvement for the problem.