If sys-usb powered off computer shuts down after resuming from suspend

[lead4good]

Qubes OS version:

R4.0 rc5

Affected component(s):

Resume after suspend

---

Steps to reproduce the behavior:

If sys-usb is turned off suspending works. However waking after S3 Sleep leaves the screen blank and after a few seconds the computer turns off. If sys-usb is started resume after S3 Sleep works without problems.

Expected behavior:

Resume after suspend should work without sys-usb started.

Actual behavior:

Computer turns off

General notes:

• xiaomi notebook pro 15
• Kabylake R u8550u
• usb 2.0, 3.0 and 3.1

lspci.txt
lsusb.txt

[marmarek]

Do you have, by a chance, a serial console there? I guess there is some kernel panic, but it would be very helpful to get actual message.

[marmarek]

Do you have, by a chance, a serial console there? I guess there is some kernel panic, but it would be very helpful to get actual message.

[lead4good]

Unfortunately I'm not able to attach a serial control, sorry. However there is another issue with resuming from suspend state which I described in #3705. I think they could be related.

[lead4good]

Unfortunately I'm not able to attach a serial control, sorry. However there is another issue with resuming from suspend state which I described in #3705. I think they could be related.

[marmarek]

How do you shutdown sys-usb? qvm-shutdown / domains widget? poweroff from inside? qvm-kill ? Or maybe do not start it at all from the beginning?

[marmarek]

How do you shutdown sys-usb? qvm-shutdown / domains widget? poweroff from inside? qvm-kill ? Or maybe do not start it at all from the beginning?

[marmarek]

nvm, reproduced similar problem using qvm-shutdown. In my case (Thinkpad T460p, Skylake) system doesn't fully wakeup - keyboard lights turns on (FnLk, backlight), but power button keep flashing and screen stays black (no backlight either).

[marmarek]

nvm, reproduced similar problem using qvm-shutdown. In my case (Thinkpad T460p, Skylake) system doesn't fully wakeup - keyboard lights turns on (FnLk, backlight), but power button keep flashing and screen stays black (no backlight either).

[evilaliv3]

@marmarek: check what i annotated on #3901

I've the same issue on a Thinkpad T480.
The issue is solved if shutdown sys-usb before sleep and if i restart it manually after.
The issue is partially solved if i perform rrmpod xhci_pci in sys-usb before performing sleep; in this second case the system will freeze if i will perform modeprobe xhci_pci but i manage to get awake and be able to log back in xscreen.

[evilaliv3]

@marmarek: check what i annotated on #3901

I've the same issue on a Thinkpad T480.
The issue is solved if shutdown sys-usb before sleep and if i restart it manually after.
The issue is partially solved if i perform rrmpod xhci_pci in sys-usb before performing sleep; in this second case the system will freeze if i will perform modeprobe xhci_pci but i manage to get awake and be able to log back in xscreen.

[lead4good]

@evilaliv3
Can't reproduce this on my machine. Both stopping sys-usb vm and running rmmod xhci_pci only lead to the system freezing during suspend resume.

[lead4good]

@evilaliv3
Can't reproduce this on my machine. Both stopping sys-usb vm and running rmmod xhci_pci only lead to the system freezing during suspend resume.

[esote]

I am having nearly this exact problem: with sys-usb running I am able to use suspend flawlessly, however if sys-usb is shut down, I cannot suspend (cannot recover, machine reboots within seconds).

[esote]

I am having nearly this exact problem: with sys-usb running I am able to use suspend flawlessly, however if sys-usb is shut down, I cannot suspend (cannot recover, machine reboots within seconds).

[marmarek]

@evilaliv3

The issue is partially solved if i perform rrmpod xhci_pci in sys-usb before performing sleep;

Something equivalent is done automatically already (see /usr/lib/qubes/prepare-suspend and /etc/qubes-suspend-module-blacklist). So, you say you get different result if you do that manually first?

[marmarek]

@evilaliv3

The issue is partially solved if i perform rrmpod xhci_pci in sys-usb before performing sleep;

Something equivalent is done automatically already (see /usr/lib/qubes/prepare-suspend and /etc/qubes-suspend-module-blacklist). So, you say you get different result if you do that manually first?

[evilaliv3]

@marmarek: i think my issue was not related to this ticket as i was initially thinking.

there are users that (in this ticket) are reporting that they fail to perform a resume if sys-usb is not active at the sime of the suspend.

in my situation instead is the opposite ( as other thinkpad users, and as you I guess) i' having a failure on the follwing condition:

• sys-usb up
• ahci and related modules are loaded
• USB 3 is attached to sys-usb

I've three possible different fixes for the above but all require manual user action and are noisy:

• one can simply stop sys-usb manually before suspending and turn it on after resume (this is the most simple, and one can just simply keep sys-usb off as a good practice in general)
• one can manually unload the module from sys-usb: the main issue is that reloading will trigger the freeze regardles the use do it manually or the resume process uses the blacklisted modules list to reload the module (this make me think that a particular reset is necessary and should be performed before loading the ahci module)
• one can simply detach the usb3 controller from sys-usb before and after suspending.

Please let me know if you need some additional information; all that i'm reporting is for a T480

[evilaliv3]

@marmarek: i think my issue was not related to this ticket as i was initially thinking.

there are users that (in this ticket) are reporting that they fail to perform a resume if sys-usb is not active at the sime of the suspend.

in my situation instead is the opposite ( as other thinkpad users, and as you I guess) i' having a failure on the follwing condition:

• sys-usb up
• ahci and related modules are loaded
• USB 3 is attached to sys-usb

I've three possible different fixes for the above but all require manual user action and are noisy:

• one can simply stop sys-usb manually before suspending and turn it on after resume (this is the most simple, and one can just simply keep sys-usb off as a good practice in general)
• one can manually unload the module from sys-usb: the main issue is that reloading will trigger the freeze regardles the use do it manually or the resume process uses the blacklisted modules list to reload the module (this make me think that a particular reset is necessary and should be performed before loading the ahci module)
• one can simply detach the usb3 controller from sys-usb before and after suspending.

Please let me know if you need some additional information; all that i'm reporting is for a T480

[maertsen]

I experience exactly the same behaviour on my T480. I can confirm that the workarounds as described by evilaliv3 work. The immediate freeze upon loading xhci_pci post resume seems suspect, though I have no way of further debugging this.

I am open to testing things.

[maertsen]

I experience exactly the same behaviour on my T480. I can confirm that the workarounds as described by evilaliv3 work. The immediate freeze upon loading xhci_pci post resume seems suspect, though I have no way of further debugging this.

I am open to testing things.

[dmoerner]

I can reproduce this on a 3rd generation X1 Carbon, with both 4.14.18 and 4.14.35 kernels. Like @marmarek, I can see that the suspend is faulty. Keyboard lights stay on, but the power button still blinks and the screen stays off.

Like @lead4good and unlike others in the thread, shutting down sys-usb and removing the xhci_pci module from sys-usb do NOT allow the system to resume from suspend. Even if I remove the USB controllers from sys-usb, I get the exact same failure.

[dmoerner]

I can reproduce this on a 3rd generation X1 Carbon, with both 4.14.18 and 4.14.35 kernels. Like @marmarek, I can see that the suspend is faulty. Keyboard lights stay on, but the power button still blinks and the screen stays off.

Like @lead4good and unlike others in the thread, shutting down sys-usb and removing the xhci_pci module from sys-usb do NOT allow the system to resume from suspend. Even if I remove the USB controllers from sys-usb, I get the exact same failure.

[marmarek]

I can no longer reproduce it with Xen 4.8.3-8, so appears to be fixed as a side effect of recent patches.

[marmarek]

I can no longer reproduce it with Xen 4.8.3-8, so appears to be fixed as a side effect of recent patches.

[dmoerner]

Unfortunately, I can still reproduce it with Xen 4.8.3-8 in dom0 and Fedora 28 sys-usb on an X1 Carbon 3rd gen. This a completely fresh install, fully updated with current-testing. Turning off sys-usb, or disabling it entirely, does not prevent the crash on resume.

There's nothing useful in the log before suspend, it looks like, plus no error afterwards. (I did test with music playing, and this is not just a problem with the backlight and the screen - there is some sort of crash which turns on the keyboard mute light, and turns on the fans, but does not turn on the keyboard backlight, or get the system to a state where music continues to play.)

May 22 13:57:29 dom0 systemd[1]: Starting Qubes suspend hooks...
May 23 13:57:31 dom0 qmemman.daemon.algo[1784]: balance_when_enough_memory(xen_free_memory=10515436, total_mem_pref=2513697292.8, total_available_memory=5040432190.200001)
May 23 13:57:31 dom0 52qubes-pause-vms[4444]: 0
May 23 13:57:31 dom0 systemd[1]: Started Qubes suspend hooks.
May 23 13:57:31 dom0 kernel: audit: type=1130 audit(1527098251.090:174): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 23 13:57:31 dom0 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 23 13:57:31 dom0 systemd[1]: Reached target Sleep.
May 23 13:57:31 dom0 systemd[1]: Starting Suspend...
May 23 13:57:31 dom0 systemd-sleep[4457]: Suspending system...
May 23 13:57:31 dom0 kernel: PM: suspend entry (deep)

It seems that there are two bugs here - one that went away when you turn off sys-usb, affecting T-series, and which was fixed by Xen 4.8.3-8, and one that never went away, and appears to not be fixed, and which affected me, and probably @lead4good. @lead4good , is the problem fixed for you with Xen 4.8.3-8 from current-testing?

[dmoerner]

Unfortunately, I can still reproduce it with Xen 4.8.3-8 in dom0 and Fedora 28 sys-usb on an X1 Carbon 3rd gen. This a completely fresh install, fully updated with current-testing. Turning off sys-usb, or disabling it entirely, does not prevent the crash on resume.

There's nothing useful in the log before suspend, it looks like, plus no error afterwards. (I did test with music playing, and this is not just a problem with the backlight and the screen - there is some sort of crash which turns on the keyboard mute light, and turns on the fans, but does not turn on the keyboard backlight, or get the system to a state where music continues to play.)

May 22 13:57:29 dom0 systemd[1]: Starting Qubes suspend hooks...
May 23 13:57:31 dom0 qmemman.daemon.algo[1784]: balance_when_enough_memory(xen_free_memory=10515436, total_mem_pref=2513697292.8, total_available_memory=5040432190.200001)
May 23 13:57:31 dom0 52qubes-pause-vms[4444]: 0
May 23 13:57:31 dom0 systemd[1]: Started Qubes suspend hooks.
May 23 13:57:31 dom0 kernel: audit: type=1130 audit(1527098251.090:174): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 23 13:57:31 dom0 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 23 13:57:31 dom0 systemd[1]: Reached target Sleep.
May 23 13:57:31 dom0 systemd[1]: Starting Suspend...
May 23 13:57:31 dom0 systemd-sleep[4457]: Suspending system...
May 23 13:57:31 dom0 kernel: PM: suspend entry (deep)

It seems that there are two bugs here - one that went away when you turn off sys-usb, affecting T-series, and which was fixed by Xen 4.8.3-8, and one that never went away, and appears to not be fixed, and which affected me, and probably @lead4good. @lead4good , is the problem fixed for you with Xen 4.8.3-8 from current-testing?

[dmoerner]

Possibly relevant xen-devel thread, https://lists.xenproject.org/archives/html/xen-devel/2018-04/msg00794.html. I see that upstream commits cb2a4a4, 710a8eb, bb502a8 are now in Xen 4.8.3-8, so I'm stumped.

[dmoerner]

Possibly relevant xen-devel thread, https://lists.xenproject.org/archives/html/xen-devel/2018-04/msg00794.html. I see that upstream commits cb2a4a4, 710a8eb, bb502a8 are now in Xen 4.8.3-8, so I'm stumped.

[marmarek]

Possibly relevant xen-devel thread, https://lists.xenproject.org/archives/html/xen-devel/2018-04/msg00794.html.

This one was already fixed in 4.8.3-5. But there is another discussion

Can you try adding xpti=0 xen option?

[marmarek]

Possibly relevant xen-devel thread, https://lists.xenproject.org/archives/html/xen-devel/2018-04/msg00794.html.

This one was already fixed in 4.8.3-5. But there is another discussion

Can you try adding xpti=0 xen option?

[HW42]

But there is another discussion

Can you try adding xpti=0 xen option?

Given that we don't have the commit which introduced the problem it's unlikely it's this issue. But testing with xpti=off doesn't hurt.

[HW42]

But there is another discussion

Can you try adding xpti=0 xen option?

Given that we don't have the commit which introduced the problem it's unlikely it's this issue. But testing with xpti=off doesn't hurt.

[dmoerner]

Unfortunately, I still can't resume with xpti=false nor with xpti=false bti=false. (I'm adding these to the options= line in /boot/efi/EFI/qubes/xen.cfg for the relevant kernel.)

[dmoerner]

Unfortunately, I still can't resume with xpti=false nor with xpti=false bti=false. (I'm adding these to the options= line in /boot/efi/EFI/qubes/xen.cfg for the relevant kernel.)

[esote]

After updating to 4.8.3-8 (current-testing in dom0 and VMs), I still cannot recover from suspend when sys-usb is shut down (works fine when sys-usb is running).

@marmarek Quick question: why does /sys/devices/system/cpu/vulnerabilities/meltdown show "vulnerable" when in VMs it shows "Mitigation: PTI"? Is there a Xen equivalent to show Meltdown and Spectre vulnerability status?

[esote]

After updating to 4.8.3-8 (current-testing in dom0 and VMs), I still cannot recover from suspend when sys-usb is shut down (works fine when sys-usb is running).

@marmarek Quick question: why does /sys/devices/system/cpu/vulnerabilities/meltdown show "vulnerable" when in VMs it shows "Mitigation: PTI"? Is there a Xen equivalent to show Meltdown and Spectre vulnerability status?

[marmarek]

@marmarek Quick question: why does /sys/devices/system/cpu/vulnerabilities/meltdown show "vulnerable" when in VMs it shows "Mitigation: PTI"?

Linux-level mitigations are disabled in dom0 (does not apply to this type of VM).

Is there a Xen equivalent to show Meltdown and Spectre vulnerability status?

In 4.8.3-8 there is added info in xl dmesg, you should have "XPTI: enabled" somewhere there.

[marmarek]

@marmarek Quick question: why does /sys/devices/system/cpu/vulnerabilities/meltdown show "vulnerable" when in VMs it shows "Mitigation: PTI"?

Linux-level mitigations are disabled in dom0 (does not apply to this type of VM).

Is there a Xen equivalent to show Meltdown and Spectre vulnerability status?

In 4.8.3-8 there is added info in xl dmesg, you should have "XPTI: enabled" somewhere there.

[esote]

@marmarek It shows

(XEN) Speculative mitigation facilities:
(XEN)   Compiled-in support: INDIRECT_THUNK
(XEN) BTI mitigations: Thunk RETPOLINE, Others: RSB_NATIVE RSB_VMEXIT

But sudo xl dmesg | grep -i xpti returns nothing. From what I can tell, the branch target injection protections (ie retpoline) are for spectre not meltdown. I am using 4.8.3-8.

[esote]

@marmarek It shows

(XEN) Speculative mitigation facilities:
(XEN)   Compiled-in support: INDIRECT_THUNK
(XEN) BTI mitigations: Thunk RETPOLINE, Others: RSB_NATIVE RSB_VMEXIT

But sudo xl dmesg | grep -i xpti returns nothing. From what I can tell, the branch target injection protections (ie retpoline) are for spectre not meltdown. I am using 4.8.3-8.

[marmarek]

Are you sure that 4.8.3-8 is started? See at the top of xl dmesg.

[marmarek]

Are you sure that 4.8.3-8 is started? See at the top of xl dmesg.

[esote]

@marmarek You're right, it still shows 4.8.3-3. How do I go about fixing that? Reinstalling the package did not fix xl dmesg reporting it as 4.8.3-3.

[esote]

@marmarek You're right, it still shows 4.8.3-3. How do I go about fixing that? Reinstalling the package did not fix xl dmesg reporting it as 4.8.3-3.

[marmarek]

Have you applied this uefi workaround? If so, you need to copy xen.efi manually after update.

[marmarek]

Have you applied this uefi workaround? If so, you need to copy xen.efi manually after update.

[esote]

@marmarek Yes, I verified /boot/efi/EFI/qubes/xen.efi and /boot/efi/EFI/BOOT/BOOTX64.efi have the same hash (along with xen-4.8.3.efi in both /qubes and /BOOT directories).

[esote]

@marmarek Yes, I verified /boot/efi/EFI/qubes/xen.efi and /boot/efi/EFI/BOOT/BOOTX64.efi have the same hash (along with xen-4.8.3.efi in both /qubes and /BOOT directories).

[marmarek]

Hmm, have you checked xen-hypervisor package version?

[marmarek]

Hmm, have you checked xen-hypervisor package version?

[esote]

@marmarek They all are 4.8.3-8:

$ sudo dnf list installed | grep xen
libvirt-daemon-driver-xen.x86_64       3.3.0-7.fc25           @anaconda         
libvirt-daemon-xen.x86_64              3.3.0-7.fc25           @anaconda         
python3-xen.x86_64                     2001:4.8.3-8.fc25      @qubes-dom0-cached
qubes-libvchan-xen.x86_64              4.0.2-1.fc25           @qubes-dom0-cached
xen.x86_64                             2001:4.8.3-8.fc25      @qubes-dom0-cached
xen-hvm.x86_64                         2001:4.8.3-8.fc25      @qubes-dom0-cached
xen-hvm-stubdom-linux.x86_64           1.0.7-1.fc25           @anaconda         
xen-hypervisor.x86_64                  2001:4.8.3-8.fc25      @qubes-dom0-cached
xen-libs.x86_64                        2001:4.8.3-8.fc25      @qubes-dom0-cached
xen-licenses.x86_64                    2001:4.8.3-8.fc25      @qubes-dom0-cached
xen-runtime.x86_64                     2001:4.8.3-8.fc25      @qubes-dom0-cached

Should I move this into a separate issue?

[esote]

@marmarek They all are 4.8.3-8:

$ sudo dnf list installed | grep xen
libvirt-daemon-driver-xen.x86_64       3.3.0-7.fc25           @anaconda         
libvirt-daemon-xen.x86_64              3.3.0-7.fc25           @anaconda         
python3-xen.x86_64                     2001:4.8.3-8.fc25      @qubes-dom0-cached
qubes-libvchan-xen.x86_64              4.0.2-1.fc25           @qubes-dom0-cached
xen.x86_64                             2001:4.8.3-8.fc25      @qubes-dom0-cached
xen-hvm.x86_64                         2001:4.8.3-8.fc25      @qubes-dom0-cached
xen-hvm-stubdom-linux.x86_64           1.0.7-1.fc25           @anaconda         
xen-hypervisor.x86_64                  2001:4.8.3-8.fc25      @qubes-dom0-cached
xen-libs.x86_64                        2001:4.8.3-8.fc25      @qubes-dom0-cached
xen-licenses.x86_64                    2001:4.8.3-8.fc25      @qubes-dom0-cached
xen-runtime.x86_64                     2001:4.8.3-8.fc25      @qubes-dom0-cached

Should I move this into a separate issue?

[marmarek]

I think so.

[marmarek]

I think so.

[dmoerner]

I did a bit of testing with xen 4.8.3-{3..8}. The only version for which suspend and resume works for me is 4.8.3-3. I take this to mean that patch-f97838bb-x86-Move-microcode-loading-earlier.patch in 4.8.3-4 broke resume for me, but the 3 patches meant to fix this in 4.8.3-5 did not fix the issue for me.

[dmoerner]

I did a bit of testing with xen 4.8.3-{3..8}. The only version for which suspend and resume works for me is 4.8.3-3. I take this to mean that patch-f97838bb-x86-Move-microcode-loading-earlier.patch in 4.8.3-4 broke resume for me, but the 3 patches meant to fix this in 4.8.3-5 did not fix the issue for me.

[HW42]

Does 4.8.3-8 works for you if you disable microcode loading by removing ucode=scan from the Xen command line?

If there's a BIOS with updated microcode available for you system you could also try using it.

[HW42]

Does 4.8.3-8 works for you if you disable microcode loading by removing ucode=scan from the Xen command line?

If there's a BIOS with updated microcode available for you system you could also try using it.

[dmoerner]

1. Suspend and resume works for me on 4.8.3-8 by removing ucode=scan from Xen options.
2. More importantly, suspend and resume works for me on 4.8.3-8 after flashing the latest BIOS from Lenovo. For anyone else who had this problem, I needed the new microcode in Lenovo's UEFI BIOS release 1.23 (N14ET45W).

Sorry for the trouble and thank you all of the help, this can safely be closed again.

[dmoerner]

1. Suspend and resume works for me on 4.8.3-8 by removing ucode=scan from Xen options.
2. More importantly, suspend and resume works for me on 4.8.3-8 after flashing the latest BIOS from Lenovo. For anyone else who had this problem, I needed the new microcode in Lenovo's UEFI BIOS release 1.23 (N14ET45W).

Sorry for the trouble and thank you all of the help, this can safely be closed again.

[HW42]

So there's still a issue with microcode loading and suspend. Although it seems to be somehow system specific.

@dmoerner: For you suspend was broken regardless of the state of sys-usb, right?

[HW42]

So there's still a issue with microcode loading and suspend. Although it seems to be somehow system specific.

@dmoerner: For you suspend was broken regardless of the state of sys-usb, right?

[dmoerner]

Just to be clear: After the BIOS update, I was able to suspend and resume with 4.8.3-8 and with ucode=scan in the Xen options, as in the default.

@HW42: Yes, suspend was broken before the microcode update regardless of the state of sys-usb.

[dmoerner]

Just to be clear: After the BIOS update, I was able to suspend and resume with 4.8.3-8 and with ucode=scan in the Xen options, as in the default.

@HW42: Yes, suspend was broken before the microcode update regardless of the state of sys-usb.

[HW42]

Just to be clear: After the BIOS update, I was able to suspend and resume with 4.8.3-8 and with ucode=scan in the Xen options, as in the default.

Thanks, for clarification, but I got this. This still means there's a bug with microcode loading and suspend. Which ideally should be fixes, especially since there are many system without available BIOS updates.

[HW42]

Just to be clear: After the BIOS update, I was able to suspend and resume with 4.8.3-8 and with ucode=scan in the Xen options, as in the default.

Thanks, for clarification, but I got this. This still means there's a bug with microcode loading and suspend. Which ideally should be fixes, especially since there are many system without available BIOS updates.

[lead4good]

As of 4.8.3-8 the problem persists on my hardware.

[lead4good]

As of 4.8.3-8 the problem persists on my hardware.

[Scinawa]

As of 4.8.3-7 I cannot update to -8 using qubes-dom0-update --enablerepo=qubes-dom0-current-testing xen because of an error "xen-2001:4.8.3.7.fc25"is already installed, skpping. Even if on the repo I see just 4.8.3-8. This is probably due to a issue with the versioning, and I can just wait for the update to be released in the stable version of qubes.

If I understood correctly, T480 users are meant to

• update the microcode of the bios via usb stick with the iso from here
• remove ucode=scan'' from/boot/efi/EFI/qubes/xen.cfg``

This will allow qubes to resume from the black screen while resuming from sleeping (which occured occasionally, not always). This is another problem than @evilaliv3 and other had with sys-usb which freezes., and is adressed elsewhere.

[Scinawa]

As of 4.8.3-7 I cannot update to -8 using qubes-dom0-update --enablerepo=qubes-dom0-current-testing xen because of an error "xen-2001:4.8.3.7.fc25"is already installed, skpping. Even if on the repo I see just 4.8.3-8. This is probably due to a issue with the versioning, and I can just wait for the update to be released in the stable version of qubes.

If I understood correctly, T480 users are meant to

• update the microcode of the bios via usb stick with the iso from here
• remove ucode=scan'' from/boot/efi/EFI/qubes/xen.cfg``

This will allow qubes to resume from the black screen while resuming from sleeping (which occured occasionally, not always). This is another problem than @evilaliv3 and other had with sys-usb which freezes., and is adressed elsewhere.