Dom0 to DomU window GUI pixel precision issues (i.e. some drag and drop scenarios, or some dynamically changing content scenarios)

[Aekez]

Qubes OS version:

Qubes 4.0. (final release).

@andrewdavidwong
This issue is about one of the kind of common, long-standing bug problems that makes it very odd if there is no existing issue on this problem. I'm therefore hesistent to make a new issue on this, given that the problem has been around for a very long time and has a common occurrence on Qubes systems. But because I can't find it in searches, I'll put up an issue for it. I apologize if it turns out to be a duplicate after all.

Affected component(s):

• Currently only tested problem on XFCE4.
• The code presenting screen content between virtual domain windows, and the content within the VM.
• It sometimes appear to happen more easily in some Qubes templates than others, but generally many of the below listed problems seem to happen in all templates (fedora/debian/whonix).

---

Steps to reproduce the behavior:

Some of the below listed problems are always there, other of the problems can happen from time to time (presumably triggered by something). For example the disappearance of the blinking text-cursor when writing in text or browsers, doesn't always happen and seem to only happen under special and currently unknown conditions, but in contrast the difficulty of drag and dropping bookmarks in Firefox almost always happens, around 70-90% of the time, (to clarify: when dragging a tab window down into a bookmark folder, in order to bookmark it, on the optional enabled bookmark panel).

Expected behavior:

• Apps GUI working as they would when running on bare-metal OS systems (non-virtual).
• Apps dynamic graphic changes working as they would when running on bare-metal OS systems (non-virtual).

Actual behavior:

Scenario common-factors

• The GUI bugs happen in various different programs (and sometimes in different ways too) on Qubes systems.
• Releasing windows focus (via dom0 control), seem to immediately fix many of these GUI issues in domU's.
  • some GUI issues still persist until VM is restarted.
  • other GUI issues still persist all the time, even if restarting.
  • There is also another key difference, i.e. drag and drop bookmarks issues in Firefox, is not fixed by re-focus of window.
  • Yet all these problems have a common-factor, the GUI screen bugging in VM's.

Similar scenarios

• Firefox tabs that suddenly bugs and become transparent and cannot be seen, and nothing happens if clicked on them, however they're still there, even if you click on the invisible tab, but nothing happens.
  • But if you then after clicking and nothing happens, also alt-tab the window to bring it behind and then back to front again, this will bring the tab into focus. When this start to happen, the only remedy seem to be either to keep alt-tab'ing after every time clicking on a browser tab, or restart the AppVM to make it stop doing it (at least until the bug is triggered again).
• Another problem I found that seem similar, is the typing cursor suddenly disappearing in different apps, like text editors or browsers, where the only fix is to alt-tab or minimize window, and then bringing up the app window again, at which point the cursor starts blinking correctly in the text-editor or browser.
• Popup menu issues
  • Right clicking, i.e. in different otherwise normally working apps, sometimes doesn't work at all.
  • Right clicking i.e. in different otherwise normally working apps, works, but clicking anything on the popup menu doesn't work.
  • Right clicking i.e. in different otherwise normally working apps, works, but the hover color doesn't appear when holding mouse over pop-up menu selection, even though clicking works.
    • All of these right clicking issues seem related to also happen in the Firefox tab issue (albeit does not appear to be restricted to Firefox issues), and seem to happen at the same time (and triggered by the same trigger) when the Firefox tab issue happens.
• Firefox bookmark issues
  • Drag and drop bookmarks from tab to folder in menu-bar for bookmarks.
    • This one is typically very easy to re-produce as it happens very often, fedora-26/fedora28 template.

General notes:

---

Related issues:

This section has been covered at the very top of this issue, to try make it be read first (to potentially save readers time).

[Polygonbugs]

Firefox tabs that suddenly bugs and become transparent and cannot be seen, and nothing happens if clicked on them, however they're still there, even if you click on the invisible tab, but nothing happens.

Is this qubes related? I thought it was related to firejail... If it is, then firefox sudden death is common thing to everyone. I suffered seveal times when watching youtube video. Firefox windows closed but I can still hear sound of video. Maybe related?

Another problem I found that seem similar, is the typing cursor suddenly disappearing in different apps, like text editors or browsers, where the only fix is to alt-tab or minimize window, and then bringing up the app window again, at which point the cursor starts blinking correctly in the text-editor or browser.

It is problem I'm suffering. I sometimes can't see blinking cursor when writing report on github and reddit. Also, I can't close other windows (such as nautilus) when writing document with libreoffice on different virtual desktop.

[Polygonbugs]

Firefox tabs that suddenly bugs and become transparent and cannot be seen, and nothing happens if clicked on them, however they're still there, even if you click on the invisible tab, but nothing happens.

Is this qubes related? I thought it was related to firejail... If it is, then firefox sudden death is common thing to everyone. I suffered seveal times when watching youtube video. Firefox windows closed but I can still hear sound of video. Maybe related?

Another problem I found that seem similar, is the typing cursor suddenly disappearing in different apps, like text editors or browsers, where the only fix is to alt-tab or minimize window, and then bringing up the app window again, at which point the cursor starts blinking correctly in the text-editor or browser.

It is problem I'm suffering. I sometimes can't see blinking cursor when writing report on github and reddit. Also, I can't close other windows (such as nautilus) when writing document with libreoffice on different virtual desktop.

[Aekez]

Is this qubes related? I thought it was related to firejail... If it is, then firefox sudden death is common thing to everyone. I suffered seveal times when watching youtube video. Firefox windows closed but I can still hear sound of video. Maybe related?

I think it might be related, I can't be completely sure though, but it has the common factors with the disappearing blinking text-cursor, and doesn't happen all the time, but "until some triggers it". Both of them can be temporarily fixed by alt-tabbing focus the window, which short-term fixes the missing cursor, and the transparent tabs need an alt-tab every time you click a new invisible bugged tab. So both respond to alt-tab, although the missing blinking text-cursor tend to last a bit longer before it breaks again. Both also get more permanently "fixed" upon restarting the VM, that is, until it is somehow triggered once again.

• I'm not sure which precise mechanics are involved, maybe it isn't a Qubes problem, but it would be great if we can find the exact issue.

Also, I can't close other windows (such as nautilus) when writing document with libreoffice on different virtual desktop.

Could this by any chance be while doing qvm-open-in-vm?

[Aekez]

Is this qubes related? I thought it was related to firejail... If it is, then firefox sudden death is common thing to everyone. I suffered seveal times when watching youtube video. Firefox windows closed but I can still hear sound of video. Maybe related?

I think it might be related, I can't be completely sure though, but it has the common factors with the disappearing blinking text-cursor, and doesn't happen all the time, but "until some triggers it". Both of them can be temporarily fixed by alt-tabbing focus the window, which short-term fixes the missing cursor, and the transparent tabs need an alt-tab every time you click a new invisible bugged tab. So both respond to alt-tab, although the missing blinking text-cursor tend to last a bit longer before it breaks again. Both also get more permanently "fixed" upon restarting the VM, that is, until it is somehow triggered once again.

• I'm not sure which precise mechanics are involved, maybe it isn't a Qubes problem, but it would be great if we can find the exact issue.

Also, I can't close other windows (such as nautilus) when writing document with libreoffice on different virtual desktop.

Could this by any chance be while doing qvm-open-in-vm?

[andrewdavidwong]

Possibly related issues: #3267, #738, #1455

[andrewdavidwong]

Possibly related issues: #3267, #738, #1455

[Aekez]

Possibly related issues: #3267, #738, #1455

Thanks for finding these, I'll start read them and compare them to see if I can find any cross-over relations.

About #3267

Quote from issue:

"Actually a previous focused program (like xev or the browser) receive input iff an unfocused window and the former overlap. Furthermore input is stolen until a click happens on the unfocused window."

This indeed could be related, but I'm not sure if it is. It kind of more looks like a problem in dom0 itself, rather than the code translation between graphics in dom0 and OS/Apps in domU? I won't write it off though, it does seem to be 50/50 whether it is related or not, depending on how the actual mechanics work in the affected system components between these two issues?

About #738

Quote from issue:

"At that point, with the KeePassX window in the foreground, clicking entries in it with the mouse, anything I type is received by the non-active Firefox window, not just CTRL-V."

This seems very much to be the same as #3267 issue above, and probably has the same conclusion as above if they indeed are related.

About #1455

Quote from issue:

"At step 3, window is not switched. At step 4, user interacts with Geany, despite the fact that Geany is seemingly not the current window. The Geany window is not trying to get the attention."

It seems #3267, #738, #1455 cover the same issue here? I'm still not sure if it's related to this current issue though, but it's definitely a good question. I don't have the technical insight required, but it does look to me that these 3 issues are related to the copy/paste table and windows focus problems, while this current issue is more related to pixel problems in translations between dom0 graphics rendering and the domU OS/App graphic rendering. So it seems like they are not related, but I can't be sure either.

Increasing the exact description of this issue

I'm planning to make screenshots as I encounter the issues again, in order to help making it much easier to see the actual issue. I'll put the screenshots in this issue as I incrementally gather them over time.

[Aekez]

Possibly related issues: #3267, #738, #1455

Thanks for finding these, I'll start read them and compare them to see if I can find any cross-over relations.

About #3267

Quote from issue:

"Actually a previous focused program (like xev or the browser) receive input iff an unfocused window and the former overlap. Furthermore input is stolen until a click happens on the unfocused window."

This indeed could be related, but I'm not sure if it is. It kind of more looks like a problem in dom0 itself, rather than the code translation between graphics in dom0 and OS/Apps in domU? I won't write it off though, it does seem to be 50/50 whether it is related or not, depending on how the actual mechanics work in the affected system components between these two issues?

About #738

Quote from issue:

"At that point, with the KeePassX window in the foreground, clicking entries in it with the mouse, anything I type is received by the non-active Firefox window, not just CTRL-V."

This seems very much to be the same as #3267 issue above, and probably has the same conclusion as above if they indeed are related.

About #1455

Quote from issue:

"At step 3, window is not switched. At step 4, user interacts with Geany, despite the fact that Geany is seemingly not the current window. The Geany window is not trying to get the attention."

It seems #3267, #738, #1455 cover the same issue here? I'm still not sure if it's related to this current issue though, but it's definitely a good question. I don't have the technical insight required, but it does look to me that these 3 issues are related to the copy/paste table and windows focus problems, while this current issue is more related to pixel problems in translations between dom0 graphics rendering and the domU OS/App graphic rendering. So it seems like they are not related, but I can't be sure either.

Increasing the exact description of this issue

I'm planning to make screenshots as I encounter the issues again, in order to help making it much easier to see the actual issue. I'll put the screenshots in this issue as I incrementally gather them over time.

[Aekez]

Here is the first screenshot of the drag and drop issue which appears in Firefox

• This is a fresh new fedora-28 VM for testing.
  • The bug did not trigger in this fresh new fedora-28 AppVM, I'm not sure if this is because the problem is fixed in a fresh new fedora-28 AppVM, or if it is because it has a delayed effect like some of the other similar bugs do. Anyway it makes no difference to the screenshot, because it would have to be a movie to show the exact issue here, because when it bugs you have to move the mouse over these locations back and forth in order to get the drop icon to appear, and then successfully drop the tab to bookmark. Many times it isn't enough to move the mouse back and forth on the individual locations to get it to register before dropping, in which case one can move the bookmark to one of the other 3 locations, and then drag and drop the bookmark from the panel to the correct folder (which always works).
  • Important to note that moving already existing bookmarks on the bookmark panel is never a problem, it is only and only when moving a tab to bookmark it becomes a problem as described above.'
  • However while moving non-tab bookmark around on the panel is never a problem, it is not always a success to drop it on one of the other non-folder 3 locations, which often suffer some the same problem as trying to drop it in the folder. Generally one of the locations will work though, like within 3rd or 4th attempt if unlucky.
    • It appears the inclusion of the folder aggravates the problem, when the bug triggers, it may be easier to drop the tab between folders to turn it into a bookmark, before moving it to the correct folder. In other words, it seems the more moving parts there is (folder popup), the less chance of success.
• Generally this issue, once it triggers, is a problem for folders some 70-80% of the time, while only around some 30%-50% of the time when directly on the panel itself between folders/bookmarks.
  • I'm not sure why it doesn't happen on a fresh new fedora-28 AppVM, remains to be verified if this fixes the problem or not, I'll update when I find out more.

• I'll keep putting up more screenshots or info as I gather them.

[Aekez]

Here is the first screenshot of the drag and drop issue which appears in Firefox

• This is a fresh new fedora-28 VM for testing.
  • The bug did not trigger in this fresh new fedora-28 AppVM, I'm not sure if this is because the problem is fixed in a fresh new fedora-28 AppVM, or if it is because it has a delayed effect like some of the other similar bugs do. Anyway it makes no difference to the screenshot, because it would have to be a movie to show the exact issue here, because when it bugs you have to move the mouse over these locations back and forth in order to get the drop icon to appear, and then successfully drop the tab to bookmark. Many times it isn't enough to move the mouse back and forth on the individual locations to get it to register before dropping, in which case one can move the bookmark to one of the other 3 locations, and then drag and drop the bookmark from the panel to the correct folder (which always works).
  • Important to note that moving already existing bookmarks on the bookmark panel is never a problem, it is only and only when moving a tab to bookmark it becomes a problem as described above.'
  • However while moving non-tab bookmark around on the panel is never a problem, it is not always a success to drop it on one of the other non-folder 3 locations, which often suffer some the same problem as trying to drop it in the folder. Generally one of the locations will work though, like within 3rd or 4th attempt if unlucky.
    • It appears the inclusion of the folder aggravates the problem, when the bug triggers, it may be easier to drop the tab between folders to turn it into a bookmark, before moving it to the correct folder. In other words, it seems the more moving parts there is (folder popup), the less chance of success.
• Generally this issue, once it triggers, is a problem for folders some 70-80% of the time, while only around some 30%-50% of the time when directly on the panel itself between folders/bookmarks.
  • I'm not sure why it doesn't happen on a fresh new fedora-28 AppVM, remains to be verified if this fixes the problem or not, I'll update when I find out more.

• I'll keep putting up more screenshots or info as I gather them.

[Aekez]

Doing these comparison tests for screenshots have given new insight on the problem

• There appears to be a difference between fedora-27 and fedora-28, both fully updated to this day.
  • fedora-28 does in preliminary testings appear to work correctly, at least for now. To be determined.
  • It remains unclear if the fedora-28 bugs again over a period of time once it gets triggered, to be determined.
  • It remains unclear if all the other similar issues are fixed in fedora-28 or not, to be determined.

Screenshots for the popup menu problem

• This is a comparing between fedora-27 and fedora-28, both completely fresh new AppVM's on fully updated templates. Both templates are clean from third-party apps or modifications.

fedora-27 template

fedora-28 template

Edit for further findings in relation to this comment.

• It appears the popup menu issue is semi-fixed in fedora-28, and not-fixed in fedora-27.
• Right click on folder bookmark content in fedora 28 bookmark panel works unless you right click on a bookmark or folder, located in a sub-folder of a panel-folder, at which point the bug still exists in fedora-28 under this specific condition (exactly like shown in the above screenshots).
• Right click on folder bookmark content in fedora 27 bookmark panel does not work whenever right clicking on any bookmarks or folders, in any folder located on the bookmark panel (same as screenshot up above, however the problem unlike fedora-28 also exists in the panel-folder in fedora-27).
• Both fedora-27 and fedora-28 has no issues when right clicking directly on folders or bookmarks on the panel itself, and I have never seen an issue here my self either. This problem is only in regard to right clicking on folders or bookmarks within the pop-up panel-folder (right-clicking on the contents), either the first and all sub-folders (as in fedora-27), or only the sub-folders (as in fedora-28).

Edit2: for further findings in relation to this comment.

• It appears that the bug difference between fedora-27 and fedora-28 is extended by a random trigger of sorts. After some testings with bookmarks in these fresh new AppVM's, eventually the bug also triggers in fedora-28 in sub-folders.
  • In effect this means that what is shown for the fedora-28 screenshot, becomes like shown in the fedora-27 screenshot.
    • Restarting the AppVM resets this bug in fedora-28.
    • fedora-27 has this particular right-click bug from the very start, while fedora-28 seem to attempt to fix it, but it still comes back after it is somehow triggered.

edit3: for further findings in relation to this comment and previous comments.

• At this point, given the difference between fedora-27 and fedora-28, this may be a fedora issue rather than a Qubes issue?
  • It's uncertain if all of these similar issues are like this though, for example the disappearing text-cursor does not yet have indications of being a fedora issue but remain at this point unclear in that regard.

[Aekez]

Doing these comparison tests for screenshots have given new insight on the problem

• There appears to be a difference between fedora-27 and fedora-28, both fully updated to this day.
  • fedora-28 does in preliminary testings appear to work correctly, at least for now. To be determined.
  • It remains unclear if the fedora-28 bugs again over a period of time once it gets triggered, to be determined.
  • It remains unclear if all the other similar issues are fixed in fedora-28 or not, to be determined.

Screenshots for the popup menu problem

• This is a comparing between fedora-27 and fedora-28, both completely fresh new AppVM's on fully updated templates. Both templates are clean from third-party apps or modifications.

fedora-27 template

fedora-28 template

Edit for further findings in relation to this comment.

• It appears the popup menu issue is semi-fixed in fedora-28, and not-fixed in fedora-27.
• Right click on folder bookmark content in fedora 28 bookmark panel works unless you right click on a bookmark or folder, located in a sub-folder of a panel-folder, at which point the bug still exists in fedora-28 under this specific condition (exactly like shown in the above screenshots).
• Right click on folder bookmark content in fedora 27 bookmark panel does not work whenever right clicking on any bookmarks or folders, in any folder located on the bookmark panel (same as screenshot up above, however the problem unlike fedora-28 also exists in the panel-folder in fedora-27).
• Both fedora-27 and fedora-28 has no issues when right clicking directly on folders or bookmarks on the panel itself, and I have never seen an issue here my self either. This problem is only in regard to right clicking on folders or bookmarks within the pop-up panel-folder (right-clicking on the contents), either the first and all sub-folders (as in fedora-27), or only the sub-folders (as in fedora-28).

Edit2: for further findings in relation to this comment.

• It appears that the bug difference between fedora-27 and fedora-28 is extended by a random trigger of sorts. After some testings with bookmarks in these fresh new AppVM's, eventually the bug also triggers in fedora-28 in sub-folders.
  • In effect this means that what is shown for the fedora-28 screenshot, becomes like shown in the fedora-27 screenshot.
    • Restarting the AppVM resets this bug in fedora-28.
    • fedora-27 has this particular right-click bug from the very start, while fedora-28 seem to attempt to fix it, but it still comes back after it is somehow triggered.

edit3: for further findings in relation to this comment and previous comments.

• At this point, given the difference between fedora-27 and fedora-28, this may be a fedora issue rather than a Qubes issue?
  • It's uncertain if all of these similar issues are like this though, for example the disappearing text-cursor does not yet have indications of being a fedora issue but remain at this point unclear in that regard.

[marmarek]

Firefox tabs that suddenly bugs and become transparent and cannot be seen, and nothing happens if clicked on them, however they're still there, even if you click on the invisible tab, but nothing happens.

Looks similar to #1502, #2405

[marmarek]

Firefox tabs that suddenly bugs and become transparent and cannot be seen, and nothing happens if clicked on them, however they're still there, even if you click on the invisible tab, but nothing happens.

Looks similar to #1502, #2405

[Aekez]

@marmarek

Looks similar to #1502, #2405

Cheers, I'll try give my input on any differences I can perceive between these issues and the problems in this current issue.

About #1502

This does indeed look very similar, yet it also has differences by the looks of it. I'll bullet-point below the differences between #1502 and this current issue that I can identify.

• Not all of Firefox is affected like in #1502, rather only sub-windows (like tabs) or some types of the popup-menus within Firefox window environment tends to be bugging out.
  • So anything with an environment of its own, like tab windows, pop-up menus, and actions associated with popup menus (like drag and drop tabs to bookmark). Everything else keeps working normal in Firefox, but anything with the nature of a sub-window within a bigger-window of the overall Firefox window is what bugs in one way or another. This bug with the invisible Firefox tabs is also hard to reproduce because it's unclear what triggers it, and there is often a longer period of time between them.
• The bug for invisible bugged tabs happens without full-screen.
  • This happened recently in my Whonix-ws-14 based AppVM, which I always keep in its original window size (never changing window size, let alone going full-screen). The invisible tab bug still happened under this scenario.
  • Using alt+space+f or alternatively F11 full-window keybind works smoothly when tested in this AppVM where the bug happened (as pr. time when writing this message), and doesn't seem to trigger the bug, at least not when I tried just now.
  • Mixing different full-screen modes doesn't seem to trigger it either, i.e. maximize window before pressing F11 or alt+space+f , or reverse going backwards from the different full-screen states. Maybe I missed out on a combination here, but I haven't found one here that triggers it yet.
• Bug frequency
  • There is a chance I might have accidentally gotten the window in full-screen while using my Whonix-ws-14 based AppVM. But this is only if I accidentally did it without paying attention to it and that I don't remember this because I might (and probably did) just subconsciously undo the full-screen again if I accidentally did it, without thinking much of it.
    • Whether this is the case is currently unclear to me.
  • This particular issue on invisible tabs in Firefox doesn't happen too often (probably the most rare of the bunch of similar bugs mentioned in this current issue), but it does happen from time to time, like once a month or so, I'd estimate? So it seems something must trigger it, whatever that is. But it doesn't help that memory is a bit vague here since I didn't keep observation logs on this issue, but I'll try keep it as objective as possible.
  • I do know however that the invisible firefox tab bug happened recently when I was using Whonix, which for this particular Whonix AppVM never goes into full-screen (I only use full-screen when watching streams/movies, but I never watch any streams/movies on this particular Whonix AppVM, and this particular bug also happened again recently (a couple of days ago), that's why I can remember this with reasonable certainty).

About #2405

About any similarities on missing kernel module, I unfortunately have absolutely no ideas. I haven't observed any missing modules in the AppVM's, but the VM boot doesn't present the booting state during VM start either, so I might have missed it.

• Is this something I should keep track on in order to help track this issue? Feel free to let me know if I need to start track it/log missing modules on VM's.

[Aekez]

@marmarek

Looks similar to #1502, #2405

Cheers, I'll try give my input on any differences I can perceive between these issues and the problems in this current issue.

About #1502

This does indeed look very similar, yet it also has differences by the looks of it. I'll bullet-point below the differences between #1502 and this current issue that I can identify.

• Not all of Firefox is affected like in #1502, rather only sub-windows (like tabs) or some types of the popup-menus within Firefox window environment tends to be bugging out.
  • So anything with an environment of its own, like tab windows, pop-up menus, and actions associated with popup menus (like drag and drop tabs to bookmark). Everything else keeps working normal in Firefox, but anything with the nature of a sub-window within a bigger-window of the overall Firefox window is what bugs in one way or another. This bug with the invisible Firefox tabs is also hard to reproduce because it's unclear what triggers it, and there is often a longer period of time between them.
• The bug for invisible bugged tabs happens without full-screen.
  • This happened recently in my Whonix-ws-14 based AppVM, which I always keep in its original window size (never changing window size, let alone going full-screen). The invisible tab bug still happened under this scenario.
  • Using alt+space+f or alternatively F11 full-window keybind works smoothly when tested in this AppVM where the bug happened (as pr. time when writing this message), and doesn't seem to trigger the bug, at least not when I tried just now.
  • Mixing different full-screen modes doesn't seem to trigger it either, i.e. maximize window before pressing F11 or alt+space+f , or reverse going backwards from the different full-screen states. Maybe I missed out on a combination here, but I haven't found one here that triggers it yet.
• Bug frequency
  • There is a chance I might have accidentally gotten the window in full-screen while using my Whonix-ws-14 based AppVM. But this is only if I accidentally did it without paying attention to it and that I don't remember this because I might (and probably did) just subconsciously undo the full-screen again if I accidentally did it, without thinking much of it.
    • Whether this is the case is currently unclear to me.
  • This particular issue on invisible tabs in Firefox doesn't happen too often (probably the most rare of the bunch of similar bugs mentioned in this current issue), but it does happen from time to time, like once a month or so, I'd estimate? So it seems something must trigger it, whatever that is. But it doesn't help that memory is a bit vague here since I didn't keep observation logs on this issue, but I'll try keep it as objective as possible.
  • I do know however that the invisible firefox tab bug happened recently when I was using Whonix, which for this particular Whonix AppVM never goes into full-screen (I only use full-screen when watching streams/movies, but I never watch any streams/movies on this particular Whonix AppVM, and this particular bug also happened again recently (a couple of days ago), that's why I can remember this with reasonable certainty).

About #2405

About any similarities on missing kernel module, I unfortunately have absolutely no ideas. I haven't observed any missing modules in the AppVM's, but the VM boot doesn't present the booting state during VM start either, so I might have missed it.

• Is this something I should keep track on in order to help track this issue? Feel free to let me know if I need to start track it/log missing modules on VM's.