[BUG][Wayland] Dropdown menu moves when navigated

[ngortheone]

OS: FreeBSD 14
Wayland: Sway 1.7

Dropdown menu changes it's position on the screen, after it was opened and while being navigated with the mouse.

Attached video demonstrates the issue with chart-simple example:

recording.mp4

[ManoloFLTK]

The video does not work.

[ngortheone]

😕 works for me in chromium..

Let me see if I can fix it

[ngortheone]

This should work
recording.webm

[ManoloFLTK]

Yes, the new video works now. Thanks.

That's the way FLTK menus behave under Wayland. Ultimately, the cause of this behaviour is that Wayland prohibits to know where a window is inside a display. Thus, an app cannot know what part of a popup menu window is in and what part is out of the display. The app considers its window delimits the area where menus are visible and thus moves menus around to keep the currently selected menu item inside the app's window.

This won't be changed unless information about how to know, under Wayland, what part of a popup window is visible is revealed to me.

[ngortheone]

Maybe @emersion can help with advice 🙏

[ngortheone]

copying IRC conversation with @kennylevinsen that happened in #wayland channel

ngortheone: there is no wayland-specific reason for the behavior in that video
I have not looked at what fltk currently does, but all one needs to do is
create a popup surface with the right initial position relative to its parent,
as well as proper configuration (anchor, gravity, constraint adjustment to
taste)

There is definitely no need for window position feedback to make context menus.

[ManoloFLTK]

The above-mentionned commit should fix this issue.
@ngortheone Please report whether you confirm.

[Albrecht-S]

@ManoloFLTK The fix in 694df9d is a great achievement, thanks.

However, it has a side effect that sometimes submenus are not "opened".
Test case:

1. run test/menubar
2. open the File menu and move to Submenus: this works (submenus are opened)
3. resize the window smaller (e.g. as small as possible)
4. repeat step 2: submenu is not opened

Note that the "File" menu is not scrolled when the mouse leaves the window area (which may be the root cause of the issue?) whereas the "Huge" menu is scrolled.

This behavior is a regression of commit 694df9d. The previous behavior was that the "File" menu was scrolled when the mouse left the window borders and the submenus were opened correctly.

FWIW, I made a video:

menubar.mp4

The video demonstrates that the submenus are not opened (step 4) and how resizing the underlying window changes the behavior as soon as the respective menu item is inside the window.

[ManoloFLTK]

Note that the "File" menu is not scrolled when the mouse leaves the window area (which may be the root cause of the issue?)

This menu positioning is quite difficult. I believe the cause of the problem you report is:
These positioner-created rules are constrained by the requirement that a child surface must intersect with or be at least partially adjacent to its parent surface.
as mentionned there.

[ngortheone]

@kennylevinsen in #wayland channel:

the issue is likely that the second popup is attempted created from the top level (and thus out of bounds) when it should be a child of the first popup

[ManoloFLTK]

Commit e73b2da should fix the issue with submenus.

[Albrecht-S]

Commit e73b2da should fix the issue with submenus.

Thanks, this is much better but there's still some strange behavior in test/menubar:

1. Run test/menubar in its original size.
2. Open the File menu, scroll down to Submenus
3. Move the mouse over submenu and further to item 1 - item 4: OK, the submenus are all opened outside the main window and can be navigated as expected
4. Move back over the first submenu and down to after submenu: the first submenu window "jumps up" which is unexpected.
5. Further navigation over the second submenu (item 1 - item 4) works again as expected.

The following video demonstrates this behavior.

menubar_e73b2da5e.mp4

Further observation: when resizing the main window height some pixels smaller the first submenu "jumps" already if and only if the "submenu" item of the first submenu is (at least partially) outside the main window area. This doesn't seem intended, particularly because the second level submenu works as expected. Or maybe there's a reason I don't see...

[ManoloFLTK]

@Albrecht-S That's the way it is. Please read the source file comment "Implementation note" of commit e73b2da. This is a working implementation, not a perfect implementation which would require, I believe, to completely rework the handling of menu windows by FLTK, because FLTK and Wayland approaches to popup window positioning are extremely different.
The "File" menu contains a submenu. It's therefore handled (that's the new stuff in e73b2da) as before commit 694df9d. In that situation, popups are positioned so that the currently selected menu item remains inside the parent window, to ensure Wayland accepts to open a popup next to it, and to also ensure the item is visible provided the parent window itself is visible. Moving from an item close to the window bottom to the next item makes the menu move up.
In contrast, menus (they can be submenus) that don't contain submenus are processed the Wayland way (that's the new stuff in 694df9d), which ensures they won't expand beyond the display limits and allows them to expand beyond the window limits. In this situation, FLTK can navigate up and down a menu without it jumping up or down even if the selected item is outside the parent window. That's the case in your video of the item 1/2/3/4 sub-sub-menu. Another "bug" is that when a popup is moved up, the small menutitle window above it disappears, covered by the menuwindow.

[ngortheone]

The original issue is fixed, should we close this issue?

[ManoloFLTK]

On my side, I would agree to see this issue closed. Let's see what's @Albrecht-S 's view.

[Albrecht-S]

@ManoloFLTK I read the "implementation note" mentioned by you, and I understand what you wrote (I missed the "submenus exception" previously). Given that note, I see that the behavior is as described. If this is the best we can get w/o reworking the entire FLTK menu handling, then I agree that the issue is fixed and can be closed.

But I have one remaining question: why exactly do we need to exclude menus with submenus from the "Wayland way" of positioning menus?

@ngortheone wrote above (citing someone else): "the issue is likely that the second popup is attempted created from the top level (and thus out of bounds) when it should be a child of the first popup", and you (@ManoloFLTK) wrote before "These positioner-created rules are constrained by the requirement that a child surface must intersect with or be at least partially adjacent to its parent surface". If I take these two statements together I would think that we can open each submenu as a "child of" the previous (sub)menu, which would always be "adjacent to its parent surface". I tried hard to understand what the current code does and tried to modify it (by educated guessing and trial and error) but I don't understand what exactly is going on - and I would need too much time to try to understand the Wayland stuff that's involved.

I assume that the answer to my question is that the current logic of opening menu windows doesn't allow to use the previous menu window as the "parent" of the next submenu window. Is this correct? What I see (in the debugger) is that each menu is opened relatively to the main window (if I understand this correctly). I would guess that the previous (i.e. current) menu window is always Fl::first_window() when a new menu is opened, but I believe that you must have tried this as well.

Can you shed some light on the question why we can't open the "next" menu window relatively to the "current" menu window?

Please don't understand me wrong, I don't insist on a "perfect" menu system if this is all we can get. I'd only like to understand why this is so. Thanks.

[ManoloFLTK]

@Albrecht-S All what you wrote is correct. It would be enough to make a submenu the wayland child of the previous menu for wayland rules to work. There are complications with the left display border and menu popups, though. You've also correctly seen that presently, all menus are made children of the main window. The difficulty is to change the code in Fl_Menu.cxx to catch the adequate menuwindow object and coordinates. To tell the truth, I'm progressing on this topic (see branch wayland-menu of my FLTK fork), but there are so many traps to consider…

[Albrecht-S]

@ManoloFLTK Thank you very much for these clarifications, and thank you for working further on this problem in the background. Looking forward to your results.