Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8267385: Create NSAccessibilityElement implementation for JavaComponentAccessibility #4412

Closed
wants to merge 15 commits into from
Closed
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
@@ -34,11 +34,13 @@
import java.awt.event.KeyEvent;
import java.beans.PropertyChangeEvent;
import java.beans.PropertyChangeListener;
import java.lang.annotation.Native;
import java.lang.reflect.InvocationTargetException;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.Callable;
import java.util.Arrays;

import javax.accessibility.Accessible;
import javax.accessibility.AccessibleAction;
@@ -57,6 +59,8 @@
import javax.swing.JLabel;
import javax.swing.JMenuItem;
import javax.swing.JTextArea;
import javax.swing.JList;
import javax.swing.JTree;
import javax.swing.KeyStroke;

import sun.awt.AWTAccessor;
@@ -116,12 +120,7 @@ public void propertyChange(final PropertyChangeEvent evt) {
private native void focusChanged();

static <T> T invokeAndWait(final Callable<T> callable, final Component c) {
if (c != null) {
try {
return LWCToolkit.invokeAndWait(callable, c);
} catch (final Exception e) { e.printStackTrace(); }
}
return null;
return invokeAndWait(callable, c, null);
}

static <T> T invokeAndWait(final Callable<T> callable, final Component c, final T defValue) {
@@ -551,6 +550,10 @@ public void run() {
if (pac == null) return;
AccessibleSelection as = pac.getAccessibleSelection();
if (as == null) return;
if (parent instanceof JList) {
((JList) parent).setSelectedIndex(i);
return;
}
Copy link
Member

@mrserb mrserb Jun 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the a11y interface miss "setSelectedIndex" method? The code above will not work for the custom component which uses the "AccessibleJListChild" a11y object?

Copy link
Contributor Author

@savoptik savoptik Jul 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, the current a11y interface does not allow this action.
As for the custom AccessibleJListChild, I prepared a sample project in which I implemented a custom child. The experiment showed that there are no negative consequences. The custom child works as intended.

Copy link
Member

@mrserb mrserb Aug 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the example, you use a JList class, and the checks in the code above works, but if some other "AccessibleJList" class will be used then it will not work, right? This is the reason why implementations of a11y classes usually use specific interfaces like "AccessibleJList".

Copy link
Contributor Author

@savoptik savoptik Aug 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this will not work with a custom list not inherited from JList. Unfortunately AccessibleJList is not perfect and the functionality you need is not there.
If you have a better solution, please share it. - Could you please advice on what we can do here?

Copy link
Member

@mrserb mrserb Aug 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I meant in the first message, "a11y interface miss setSelectedIndex method". I suggest filing a separate issue to accumulate such use cases and then fix them by adding additional methods to a11y interfaces. Probably something similar will be found in the Table/etc.

Copy link
Contributor Author

@savoptik savoptik Aug 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. JDK-8271846

as.addAccessibleSelection(i);
}
}, c);
@@ -651,77 +654,148 @@ public boolean[] call() throws Exception {

// Duplicated from JavaComponentAccessibility
// Note that values >=0 are indexes into the child array
static final int JAVA_AX_ALL_CHILDREN = -1;
static final int JAVA_AX_SELECTED_CHILDREN = -2;
static final int JAVA_AX_VISIBLE_CHILDREN = -3;
@Native static final int JAVA_AX_ALL_CHILDREN = -1;
@Native static final int JAVA_AX_SELECTED_CHILDREN = -2;
@Native static final int JAVA_AX_VISIBLE_CHILDREN = -3;

// Each child takes up two entries in the array: one for itself and one for its role
public static Object[] getChildrenAndRoles(final Accessible a, final Component c, final int whichChildren, final boolean allowIgnored) {
if (a == null) return null;
return invokeAndWait(new Callable<Object[]>() {
public Object[] call() throws Exception {
ArrayList<Object> childrenAndRoles = new ArrayList<Object>();
_addChildren(a, whichChildren, allowIgnored, childrenAndRoles);

/* In the case of fetching a selection, need to check to see if
* the active descendant is at the beginning of the list. If it
* is not it needs to be moved to the beginning of the list so
* VoiceOver will annouce it correctly. The list returned
* from Java is always in order from top to bottom, but when shift
* selecting downward (extending the list) or multi-selecting using
* the VO keys control+option+command+return the active descendant
* is not at the top of the list in the shift select down case and
* may not be in the multi select case.
*/
if (whichChildren == JAVA_AX_SELECTED_CHILDREN) {
if (!childrenAndRoles.isEmpty()) {
AccessibleContext activeDescendantAC =
CAccessible.getActiveDescendant(a);
if (activeDescendantAC != null) {
String activeDescendantName =
activeDescendantAC.getAccessibleName();
AccessibleRole activeDescendantRole =
activeDescendantAC.getAccessibleRole();
// Move active descendant to front of list.
// List contains pairs of each selected item's
// Accessible and AccessibleRole.
ArrayList<Object> newArray = new ArrayList<Object>();
int count = childrenAndRoles.size();
Accessible currentAccessible = null;
AccessibleContext currentAC = null;
String currentName = null;
AccessibleRole currentRole = null;
for (int i = 0; i < count; i+=2) {
// Is this the active descendant?
currentAccessible = (Accessible)childrenAndRoles.get(i);
currentAC = currentAccessible.getAccessibleContext();
currentName = currentAC.getAccessibleName();
currentRole = (AccessibleRole)childrenAndRoles.get(i+1);
if (currentName != null && currentName.equals(activeDescendantName) &&
currentRole.equals(activeDescendantRole) ) {
newArray.add(0, currentAccessible);
newArray.add(1, currentRole);
} else {
newArray.add(currentAccessible);
newArray.add(currentRole);
}
}
childrenAndRoles = newArray;
return getChildrenAndRolesImpl(a, c, whichChildren, allowIgnored);
}
}, c);
}

private static Object[] getChildrenAndRolesImpl(final Accessible a, final Component c, final int whichChildren, final boolean allowIgnored) {
if (a == null) return null;

ArrayList<Object> childrenAndRoles = new ArrayList<Object>();
_addChildren(a, whichChildren, allowIgnored, childrenAndRoles);

/* In case of fetching a selection, we need to check if
* the active descendant is at the beginning of the list, or
* otherwise move it, so that VoiceOver announces it correctly.
* The java list is always in order from top to bottom, but when
* (1) shift-selecting downward (extending the list) or (2) multi-selecting with
* the VO keys (CTRL+ALT+CMD+RETURN) the active descendant
* is not at the top of the list in the 1st case and may not be in the 2nd.
*/
if (whichChildren == JAVA_AX_SELECTED_CHILDREN) {
if (!childrenAndRoles.isEmpty()) {
AccessibleContext activeDescendantAC =
CAccessible.getActiveDescendant(a);
if (activeDescendantAC != null) {
String activeDescendantName =
activeDescendantAC.getAccessibleName();
AccessibleRole activeDescendantRole =
activeDescendantAC.getAccessibleRole();
// Move active descendant to front of list.
// List contains pairs of each selected item's
// Accessible and AccessibleRole.
ArrayList<Object> newArray = new ArrayList<Object>();
int count = childrenAndRoles.size();
Accessible currentAccessible = null;
AccessibleContext currentAC = null;
String currentName = null;
AccessibleRole currentRole = null;
for (int i = 0; i < count; i += 2) {
// Is this the active descendant?
currentAccessible = (Accessible) childrenAndRoles.get(i);
currentAC = currentAccessible.getAccessibleContext();
currentName = currentAC.getAccessibleName();
currentRole = (AccessibleRole) childrenAndRoles.get(i + 1);
if (currentName != null && currentName.equals(activeDescendantName) &&
currentRole.equals(activeDescendantRole)) {
newArray.add(0, currentAccessible);
newArray.add(1, currentRole);
} else {
newArray.add(currentAccessible);
newArray.add(currentRole);
}
}
childrenAndRoles = newArray;
}
}
}

if ((whichChildren < 0) || (whichChildren * 2 >= childrenAndRoles.size())) {
return childrenAndRoles.toArray();
}

return new Object[]{childrenAndRoles.get(whichChildren * 2), childrenAndRoles.get((whichChildren * 2) + 1)};
}

// This method is called from the native
// Each child takes up three entries in the array: one for itself, one for its role, and one for the recursion level
private static Object[] getChildrenAndRolesRecursive(final Accessible a, final Component c, final int whichChildren, final boolean allowIgnored, final int level) {
if (a == null) return null;
return invokeAndWait(new Callable<Object[]>() {
public Object[] call() throws Exception {
ArrayList<Object> currentLevelChildren = new ArrayList<Object>();
ArrayList<Object> allChildren = new ArrayList<Object>();
ArrayList<Accessible> parentStack = new ArrayList<Accessible>();
parentStack.add(a);
ArrayList<Integer> indexses = new ArrayList<Integer>();
Integer index = 0;
int currentLevel = level;
while (!parentStack.isEmpty()) {
Accessible p = parentStack.get(parentStack.size() - 1);

currentLevelChildren.addAll(Arrays.asList(getChildrenAndRolesImpl(p, c, JAVA_AX_ALL_CHILDREN, allowIgnored)));
if ((currentLevelChildren.size() == 0) || (index >= currentLevelChildren.size())) {
if (!parentStack.isEmpty()) parentStack.remove(parentStack.size() - 1);
if (!indexses.isEmpty()) index = indexses.remove(indexses.size() - 1);
currentLevel -= 1;
currentLevelChildren.clear();
continue;
}

Accessible ca = null;
Object obj = currentLevelChildren.get(index);
if (!(obj instanceof Accessible)) {
index += 2;
currentLevelChildren.clear();
continue;
}
ca = (Accessible) obj;
Object role = currentLevelChildren.get(index + 1);
currentLevelChildren.clear();

AccessibleContext cac = ca.getAccessibleContext();
if (cac == null) {
index += 2;
continue;
}

if ((cac.getAccessibleStateSet().contains(AccessibleState.SELECTED) && (whichChildren == JAVA_AX_SELECTED_CHILDREN)) ||
(cac.getAccessibleStateSet().contains(AccessibleState.VISIBLE) && (whichChildren == JAVA_AX_VISIBLE_CHILDREN)) ||
(whichChildren == JAVA_AX_ALL_CHILDREN)) {
allChildren.add(ca);
allChildren.add(role);
allChildren.add(String.valueOf(currentLevel));
}

index += 2;

if (cac.getAccessibleStateSet().contains(AccessibleState.EXPANDED)) {
parentStack.add(ca);
indexses.add(index);
index = 0;
currentLevel += 1;
continue;
}

if ((whichChildren < 0) || (whichChildren * 2 >= childrenAndRoles.size())) {
return childrenAndRoles.toArray();
}

return new Object[] { childrenAndRoles.get(whichChildren * 2), childrenAndRoles.get((whichChildren * 2) + 1) };
return allChildren.toArray();
}
}, c);
}

private static final int JAVA_AX_ROWS = 1;
private static final int JAVA_AX_COLS = 2;
@Native private static final int JAVA_AX_ROWS = 1;
@Native private static final int JAVA_AX_COLS = 2;

public static int getTableInfo(final Accessible a, final Component c,
final int info) {
@@ -740,6 +814,23 @@ public static int getTableInfo(final Accessible a, final Component c,
}, c);
}

private static int[] getTableSelectedInfo(final Accessible a, final Component c,
final int info) {
if (a == null) return null;
return invokeAndWait(() -> {
AccessibleContext ac = a.getAccessibleContext();
AccessibleTable table = ac.getAccessibleTable();
if (table != null) {
if (info == JAVA_AX_COLS) {
return table.getSelectedAccessibleColumns();
} else if (info == JAVA_AX_ROWS) {
return table.getSelectedAccessibleRows();
}
}
return null;
}, c);
}

private static AccessibleRole getAccessibleRoleForLabel(JLabel l, AccessibleRole fallback) {
String text = l.getText();
if (text != null && text.length() > 0) {
@@ -858,4 +949,18 @@ public Long call() throws Exception {
}
}, (Component)ax);
}

private static boolean isTreeRootVisible(Accessible a, Component c) {
if (a == null) return false;

return invokeAndWait(new Callable<Boolean>() {
public Boolean call() throws Exception {
Accessible sa = CAccessible.getSwingAccessible(a);
if (sa instanceof JTree) {
return ((JTree) sa).isRootVisible();
}
return false;
}
}, c);
}
}
@@ -31,11 +31,7 @@

import javax.accessibility.Accessible;
import javax.accessibility.AccessibleContext;
import javax.swing.JProgressBar;
import javax.swing.JTabbedPane;
import javax.swing.JSlider;
import javax.swing.event.ChangeEvent;
import javax.swing.event.ChangeListener;

import static javax.accessibility.AccessibleContext.ACCESSIBLE_ACTIVE_DESCENDANT_PROPERTY;
import static javax.accessibility.AccessibleContext.ACCESSIBLE_CARET_PROPERTY;
@@ -75,6 +71,8 @@ public static CAccessible getCAccessible(final Accessible a) {
private static native void menuOpened(long ptr);
private static native void menuClosed(long ptr);
private static native void menuItemSelected(long ptr);
private static native void treeNodeExpanded(long ptr);
private static native void treeNodeCollapsed(long ptr);

private Accessible accessible;

@@ -137,6 +135,13 @@ public void propertyChange(PropertyChangeEvent e) {
if (parentAccessible != null) {
parentRole = parentAccessible.getAccessibleContext().getAccessibleRole();
}

if (newValue == AccessibleState.EXPANDED) {
treeNodeExpanded(ptr);
} else if (newValue == AccessibleState.COLLAPSED) {
treeNodeCollapsed(ptr);
}

// At least for now don't handle combo box menu state changes.
// This may change when later fixing issues which currently
// exist for combo boxes, but for now the following is only