Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JDK-8306983: Do not invoke external programs when switch terminal to raw mode on selected platforms #13687

Closed
wants to merge 10 commits into from

Conversation

lahodaj
Copy link
Contributor

@lahodaj lahodaj commented Apr 27, 2023

To support JShell and other usecases, the JDK uses JLine, which provides line-editing functionality inside a terminal.

JLine has several ways to work with the terminal, based on various additional libraries and tools. Most of them are not directly usable inside the JDK, so currently, on non-Windows platforms, the JLine inside the JDK will use external programs, like stty, to inspect the terminal's properties and to switch the terminal to raw mode.

This is slightly problematic, as the external programs might be missing, and while this is not that big problem for JShell, it might be a problem for other potential uses of JLine, like using it inside System.console().

So, the proposal here is to call the corresponding native methods directly, on selected platforms for now (Linux and Mac OS/X), instead of invoking the external programs. On Windows, this has always been the case, we are using a custom implementation of the interface that maps native and Java function for JNA there, and the proposal is to do the same here. We take the appropriate mapping interface for JNA, and provide hand-written implementation for it, using JNI.

The Windows implementation is mostly unchanged, the changes are mostly non-Windows only. The overview of the changes is:

  • LastErrorException is moved from the Windows-specific code to the platform-neutral code, as it is used by all the native implementations. This is the only change that directly affects the Windows-specific code
  • unix/classes/jdk/internal/org/jline/terminal/impl/jna/JnaNativePty.java and unix/classes/jdk/internal/org/jline/terminal/impl/jna/JnaTerminalProvider.java are created based on the corresponding files from JLine. In JLine, these files directly call platform-specific implementations, but in this patch, the platform specific code is commented out, and replaced with calls to JDKNativePty, which is a new platform-specific class, that delegates to the actual platform-specific implementations. Note the JnaNativePty.java and JnaTerminalProvider.java already exist in the Windows part, but have different pieces of code commented out, which makes them substantially different.
  • for Linux and Mac OS/X, there are platform-specific implementations based on the corresponding code from JLine, with the hand-written JNI-based implementation of the JNA-based existing interfaces. They also have an implementation of JDKNativePty, which just delegates to the given platform's "NativePty" implementation.
  • the JdkConsoleProviderImpl.java has been tweaked to not allow the implementation based on the external programs, and gracefully falling back to the standard implementation. JShell is kept unchanged.

The reasons for this organization are: limiting duplication, mostly adhering to the JDK's platform specific build (which is different from the normal JLine's platform-neutral build) and limiting the difference between standard JLine and JLine inside JDK, to help future upgrades.


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed (2 reviews required, with at least 1 Reviewer, 1 Author)

Issue

  • JDK-8306983: Do not invoke external programs when switch terminal to raw mode on selected platforms

Reviewers

Contributors

  • Adam Sotona <asotona@openjdk.org>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/13687/head:pull/13687
$ git checkout pull/13687

Update a local copy of the PR:
$ git checkout pull/13687
$ git pull https://git.openjdk.org/jdk.git pull/13687/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 13687

View PR using the GUI difftool:
$ git pr show -t 13687

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/13687.diff

Webrev

Link to Webrev Comment

@lahodaj
Copy link
Contributor Author

lahodaj commented Apr 27, 2023

/contributor asotona

@bridgekeeper
Copy link

bridgekeeper bot commented Apr 27, 2023

👋 Welcome back jlahoda! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk
Copy link

openjdk bot commented Apr 27, 2023

@lahodaj Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

  • /contributor add @openjdk-bot
  • /contributor add duke
  • /contributor add J. Duke <duke@openjdk.org>

User names can only be used for users in the census associated with this repository. For other contributors you need to supply the full name and email address.

@openjdk
Copy link

openjdk bot commented Apr 27, 2023

@lahodaj The following labels will be automatically applied to this pull request:

  • build
  • kulla

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added build build-dev@openjdk.org kulla kulla-dev@openjdk.org labels Apr 27, 2023
@lahodaj
Copy link
Contributor Author

lahodaj commented Apr 27, 2023

/contributor add asotona

@openjdk
Copy link

openjdk bot commented Apr 27, 2023

@lahodaj
Contributor Adam Sotona <asotona@openjdk.org> successfully added.

@lahodaj lahodaj marked this pull request as ready for review May 3, 2023 09:29
@openjdk openjdk bot added the rfr Pull request is ready for review label May 3, 2023
@mlbridge
Copy link

mlbridge bot commented May 3, 2023

Webrevs

LDFLAGS := $(LDFLAGS_JDKLIB), \
LIBS := $(JDKLIB_LIBS) user32.lib, \
LIBS_windows := $(JDKLIB_LIBS) user32.lib, \
LIBS_unix := $(JDKLIB_LIBS) -lstdc++, \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is a C++ library, I think it's better to specify TOOLCHAIN := TOOLCHAIN_LINK_CXX than adding -lstdc++.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may also need $(LIBCXX) to force static linking when that is enabled for the build.

Copy link
Member

@naotoj naotoj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes in JdkConsoleProviderImpl.java look good to me. Didn't look at other diffs closely, but it seems some other files are missing Oracle copyright text.

Copy link
Member

@erikj79 erikj79 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build changes look good.

/reviewers 2

@openjdk
Copy link

openjdk bot commented May 5, 2023

@lahodaj This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8306983: Do not invoke external programs when switch terminal to raw mode on selected platforms

Co-authored-by: Adam Sotona <asotona@openjdk.org>
Reviewed-by: erikj, vromero, bpb

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 631 new commits pushed to the master branch:

  • 939344b: 8304685: Fix whitespace parsing in libjdwp
  • 241455f: 8307962: Exclude gc/g1/TestSkipRebuildRemsetPhase.java fails with virtual test thread factory
  • 34468e1: 8308021: Update IANA Language Subtag Registry to Version 2023-05-11
  • f0aebc8: 8305972: Update XML Security for Java to 3.0.2
  • 265f40b: 8308396: Fix offset_of conversion warnings in runtime code
  • a5343fa: 8281149: (fs) java/nio/file/FileStore/Basic.java fails with java.lang.RuntimeException: values differ by more than 1GB
  • 44218b1: 8308248: Revisit alignment of layout constants on 32-bit platforms
  • 80ef5c2: 8308410: broken compilation of test\jdk\tools\launcher\exeJliLaunchTest.c
  • 326d778: 8306057: False arguments calling dispatch_base for aarch64
  • 690d396: 8308408: Build failure with -Werror=maybe-uninitialized in libjli/java.c with GCC8
  • ... and 621 more: https://git.openjdk.org/jdk/compare/9f587d272fe7097b330d8d81b7ae43149ff92485...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready Pull request is ready to be integrated label May 5, 2023
@openjdk
Copy link

openjdk bot commented May 5, 2023

@erikj79
The total number of required reviews for this PR (including the jcheck configuration and the last /reviewers command) is now set to 2 (with at least 1 Reviewer, 1 Author).

@openjdk openjdk bot removed the ready Pull request is ready to be integrated label May 5, 2023
@@ -0,0 +1,206 @@
/*
* Copyright (c) 2015, 2018, Oracle and/or its affiliates. All rights reserved.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copyright year 2023 only if this is new.

termios data;

if (tcgetattr(fd, &data) != 0) {
jobject exc = env->NewObject(lastErrorExceptionClass,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could lines 103-106 be converted to a utility function which would then be invoked at current lines 103, 141, 163, and 197?

env->SetIntField(result, c_lflag, data.c_lflag);
env->SetIntField(result, c_line, data.c_line);
jbyteArray c_ccValue = (jbyteArray) env->GetObjectField(result, c_cc);
env->SetByteArrayRegion(c_ccValue, 0, NCCS, (signed char *) data.c_cc);//TODO: cast?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cast instead to (jbyte*) here and at lines 136 and 204?

*/
JNIEXPORT jint JNICALL Java_jdk_internal_org_jline_terminal_impl_jna_linux_CLibraryImpl_isatty
(JNIEnv *, jobject, jint fd) {
return isatty(fd);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we care if the native isatty() returns zero? Or is this dealt with somewhere else?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is only used to determine with the fd is attached to a terminal (returns 1) or not (return 0). The reasons why it is not attached to a terminal are not really important. The value is used here:
https://github.com/lahodaj/jdk/blob/4cf8f67e43f442a5c48cd30349740ac2cb638d6e/src/jdk.internal.le/unix/classes/jdk/internal/org/jline/terminal/impl/jna/JnaNativePty.java#L186

static jfieldID nativelong_value;

JNIEXPORT void JNICALL Java_jdk_internal_org_jline_terminal_impl_jna_osx_CLibraryImpl_initIDs
(JNIEnv *env, jclass) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing parameter for jclass? Maybe add clazz or unused or something?

termios data;

if (tcgetattr(fd, &data) != 0) {
jobject exc = env->NewObject(lastErrorExceptionClass,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could lines 109-112 be converted to a utility function which would then be invoked at current lines 109, 145, 167, and 197?

env->SetLongField(env->GetObjectField(result, c_cflag), nativelong_value, data.c_cflag);
env->SetLongField(env->GetObjectField(result, c_lflag), nativelong_value, data.c_lflag);
jbyteArray c_ccValue = (jbyteArray) env->GetObjectField(result, c_cc);
env->SetByteArrayRegion(c_ccValue, 0, NCCS, (signed char *) data.c_cc);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cast instead to (jbyte*) here and at lines 140 and 208?

*/
JNIEXPORT jint JNICALL Java_jdk_internal_org_jline_terminal_impl_jna_osx_CLibraryImpl_isatty
(JNIEnv *, jobject, jint fd) {
return isatty(fd);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we care if the native isatty() returns zero? Or is this dealt with somewhere else?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is only used to determine with the fd is attached to a terminal (returns 1) or not (return 0). The reasons why it is not attached to a terminal are not really important. The value is used here:
https://github.com/lahodaj/jdk/blob/4cf8f67e43f442a5c48cd30349740ac2cb638d6e/src/jdk.internal.le/unix/classes/jdk/internal/org/jline/terminal/impl/jna/JnaNativePty.java#L186

@@ -0,0 +1,210 @@
/*
* Copyright (c) 2015, 2018, Oracle and/or its affiliates. All rights reserved.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copyright year 2023 only?

import java.util.EnumSet;
import java.util.List;

//import com.sun.jna.LastErrorException;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a lot of commented code in this file, not sure if it has a purpose or just left overs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've intentionally left the original JLine code commented here and on other places, to see what the code was doing originally, and to possibly aid future merges of new versions of JLine.

@@ -0,0 +1,389 @@
/*
* Copyright (c) 2002-2020, the original author or authors.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2023 only?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the original header from JLine, which we generally keep.

@openjdk openjdk bot added the ready Pull request is ready to be integrated label May 11, 2023
Copy link
Member

@bplb bplb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C++ looks fine.

@lahodaj
Copy link
Contributor Author

lahodaj commented May 22, 2023

/integrate

@openjdk
Copy link

openjdk bot commented May 22, 2023

Going to push as commit a970519.
Since your change was applied there have been 641 commits pushed to the master branch:

  • 6b65e57: 8305785: Avoid redundant HashMap.containsKey call in java.util.regex
  • 8011ba7: 8308181: Generational ZGC: Remove CLDG_lock from old gen root scanning
  • 8aa5028: 8302344: Compiler Implementation for Unnamed patterns and variables (Preview)
  • b588797: 8307804: Reorganize ArrayJuggle test cases
  • 928fcf9: 8308000: add PopFrame support for virtual threads
  • 41beb44: 8308084: C2 fix idom bug in PhaseIdealLoop::create_new_if_for_predicate
  • b6a9f5c: 8307619: C2 failed: Not monotonic (AndI CastII LShiftI) in TestShiftCastAndNotification.java
  • eaa80ad: 8300543: Compiler Implementation for Pattern Matching for switch
  • 5ccc962: 8308342: Remove MetaspaceClosure::Ref::keep_after_pushing()
  • a0f4a94: 8307783: runtime/reflect/ReflectOutOfMemoryError.java timed out
  • ... and 631 more: https://git.openjdk.org/jdk/compare/9f587d272fe7097b330d8d81b7ae43149ff92485...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated Pull request has been integrated label May 22, 2023
@openjdk openjdk bot closed this May 22, 2023
@openjdk openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels May 22, 2023
@openjdk
Copy link

openjdk bot commented May 22, 2023

@lahodaj Pushed as commit a970519.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build build-dev@openjdk.org integrated Pull request has been integrated kulla kulla-dev@openjdk.org
6 participants