Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8254129: IR Test Framework to support regex-based matching on the IR in JTreg compiler tests #3508

Closed
wants to merge 29 commits into from

Conversation

@chhagedorn
Copy link
Member

@chhagedorn chhagedorn commented Apr 15, 2021

This RFE provides an IR test framework to perform regex-based checks on the C2 IR shape of test methods emitted by the VM flags -XX:+PrintIdeal and -XX:+PrintOptoAssembly. The framework can also be used for other non-IR matching (and non-compiler) tests by providing easy to use annotations for commonly used testing patterns and compiler control flags.

The framework is based on the ideas of the currently present IR test framework in Valhalla (mainly implemented by @TobiHartmann) which is being used with great success. This new framework aims to replace the old one in Valhalla at some point.

A detailed description about how this new IR test framework works and how it is used is provided in the README.md file and in the Javadocs written for the framework classes.

To finish a first version of this framework for JDK 17, I decided to leave some improvement possibilities and ideas to be followed up on in additional RFEs. Some ideas are mentioned in "Future Work" in README.md and were also created as subtasks of this RFE.

Testing (also described in "Internal Framework Tests in README.md):
There are various tests to verify the correctness of the test framework which can be found as JTreg tests in the tests folder. Additional testing was performed by converting all compiler Inline Types test of project Valhalla (done by @katyapav in JDK-8263024) that used the old framework to the new framework. This provided additional testing for the framework itself. We ran the converted tests with all the flag settings used in hs-tier1-9 and hs-precheckin-comp. For sanity checking, this was also done with a sample IR test in mainline.

Some stats about the framework code added to ir_framework:

  • without the Javadocs files : 60 changed files, 13212 insertions, 0 deletions.
    • without the tests and examples folder: 40 files changed, 6781 insertions
      • comments: 2399 insertions (calculated with git diff --cached !(tests|examples) | grep -c -E "(^[+-]\s*(\/)?\*)|(^[+-]\s*\/\/)")
      • which leaves 4382 lines of code inserted

Big thanks to:

  • @TobiHartmann for all his help by discussing the new framework and for providing insights from his IR test framework in Valhalla.
  • @katyapav for converting the Valhalla tests to use the new framework which found some harder to catch bugs in the framework and also some actual C2 bugs.
  • @iignatev for helping to simplify the framework usage with JTreg and with the framework internal VM calling structure.
  • and others who provided valuable feedback.

Thanks,
Christian


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issue

  • JDK-8254129: IR Test Framework to support regex-based matching on the IR in JTreg compiler tests

Reviewers

Contributors

  • Christian Hagedorn <chagedorn@openjdk.org>
  • Tobias Hartmann <thartmann@openjdk.org>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3508/head:pull/3508
$ git checkout pull/3508

Update a local copy of the PR:
$ git checkout pull/3508
$ git pull https://git.openjdk.java.net/jdk pull/3508/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3508

View PR using the GUI difftool:
$ git pr show -t 3508

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3508.diff

@bridgekeeper
Copy link

@bridgekeeper bridgekeeper bot commented Apr 15, 2021

👋 Welcome back chagedorn! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

Loading

@openjdk
Copy link

@openjdk openjdk bot commented Apr 15, 2021

@chhagedorn The following label will be automatically applied to this pull request:

  • javadoc

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

Loading

@openjdk openjdk bot added the javadoc label Apr 15, 2021
@chhagedorn
Copy link
Member Author

@chhagedorn chhagedorn commented Apr 15, 2021

/label remove javadoc

Loading

@openjdk openjdk bot removed the javadoc label Apr 15, 2021
@openjdk
Copy link

@openjdk openjdk bot commented Apr 15, 2021

@chhagedorn
The javadoc label was successfully removed.

Loading

@chhagedorn
Copy link
Member Author

@chhagedorn chhagedorn commented Apr 15, 2021

/label add hotspot-compiler
/label add hotspot

Loading

@openjdk
Copy link

@openjdk openjdk bot commented Apr 15, 2021

@chhagedorn
The hotspot-compiler label was successfully added.

Loading

@openjdk openjdk bot added the hotspot label Apr 15, 2021
@openjdk
Copy link

@openjdk openjdk bot commented Apr 15, 2021

@chhagedorn
The hotspot label was successfully added.

Loading

@chhagedorn
Copy link
Member Author

@chhagedorn chhagedorn commented Apr 15, 2021

/contributor add @TobiHartmann

Loading

@openjdk
Copy link

@openjdk openjdk bot commented Apr 15, 2021

@chhagedorn
Contributor Tobias Hartmann <thartmann@openjdk.org> successfully added.

Loading

@chhagedorn chhagedorn marked this pull request as ready for review Apr 15, 2021
@openjdk openjdk bot added the rfr label Apr 15, 2021
@mlbridge
Copy link

@mlbridge mlbridge bot commented Apr 15, 2021

Loading

@iignatev
Copy link
Member

@iignatev iignatev commented Apr 15, 2021

Hi Christian,

kudos to you, Tobias, Katya, and all involved, I have really high hopes that this framework will improve the quality of both JVM and our testing.

I'll look at the code later, a few general comments:

  • although having javadoc for testlibraries is highly desirable, I don't think we should check in the generated HTML files
  • the same goes for README.html generated from README.md
  • this library is hotspot-centric, I highly doubt that it will be used by any tests outside of the hotspot test base, hence the more appropriate location for it would be inside test/hotspot/jtreg/testlibrary.
  • I assume test/lib/jdk/test/lib/hotspot/ir_framework/tests/ are the tests for the framework, in that case they should be in test/lib-test, if we stick to test/lib as the location for the library, or in test/hotspot/jtreg/testlibrary_tests, if we move it to test/hotspot

Thanks,
-- Igor

Loading

Copy link
Contributor

@vnkozlov vnkozlov left a comment

I understand the desire to use this framework for more then IR verification but it is main goal.
Why you did not place it into test/lib/sun/hotspot/code/?

I also don't think you should include javadoc generated files into these changes.

Loading

* <li><p>{@link DontCompile @DontCompile}: Prevents any compilation of the associated helper method.</li>
* </ul>
*/
ANY(-2),
Copy link
Contributor

@vnkozlov vnkozlov Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will change (-2 -> -1) after AOT is removed (8264805).

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll change that as soon as it is integrated.

Loading

Copy link
Member

@iignatev iignatev Apr 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we also add a comment suggesting to keep in sync w/ compiler/compilerDefinitions.hpp?

Loading

* </ul>
*/
ANY(-2),
/**
Copy link
Contributor

@vnkozlov vnkozlov Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For completeness may be add value NONE(0) for Interpreter only case.

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it makes sense for completeness. But I'm not sure if this is useful to have as NONE would not be supported by the framework for the time being and would just require additional handling, informing the user that this has no effect.

Loading

/**
* Helper class to store information about a method that needs to be IR matched.
*/
class IRMethod {
Copy link
Contributor

@vnkozlov vnkozlov Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move this class IRMethod into a separate file? This file is already big.

Loading

@@ -0,0 +1,3 @@
# Minimal TEST.ROOT file to run the examples tests as if the examples would have been placed inside
Copy link
Contributor

@vnkozlov vnkozlov Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing Copyright header.

Loading

@@ -0,0 +1,12 @@
# Minimal TEST.ROOT file to run the internal framework tests as if they would have been placed inside
Copy link
Contributor

@vnkozlov vnkozlov Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing Copyright header.

Loading

Additional testing should be performed with the converted Valhalla tests (see [JDK-8263024](https://bugs.openjdk.java.net/browse/JDK-8263024)) to make sure a changeset is correct (these are part of the Valhalla CI).

Copy link
Contributor

@vnkozlov vnkozlov Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you should reference particular RFE which will be resolved eventually. The statement itself is fine.

Loading

@@ -0,0 +1,160 @@
<h1>IR Test Framework</h1>
Copy link
Member

@dougxc dougxc Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious: why HTML instead of markdown?

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also a README.md file in the same folder from which I generated this HTML file from. But as others have suggested, I will remove this html file again and only leave the markdown file.

Loading

@iignatev
Copy link
Member

@iignatev iignatev commented Apr 16, 2021

Why you did not place it into test/lib/sun/hotspot/code/?

there is an RFE to rename sun.hotspot testlibrary classes, so I wouldn't put new stuff there.

Loading

Copy link
Member

@iignatev iignatev left a comment

Hi Christian,

here is the 1st portion of my comments. I'll return to reviewing it 1st thing tomorrow...

Cheers,
-- Igor

Loading

StackWalker walker = StackWalker.getInstance(StackWalker.Option.RETAIN_CLASS_REFERENCE);
this.testClass = walker.getCallerClass();
if (VERBOSE) {
System.out.println("Test class: " + testClass);
}
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this c-tor can be replaced by:

Suggested change
StackWalker walker = StackWalker.getInstance(StackWalker.Option.RETAIN_CLASS_REFERENCE);
this.testClass = walker.getCallerClass();
if (VERBOSE) {
System.out.println("Test class: " + testClass);
}
this(StackWalker.getInstance(StackWalker.Option.RETAIN_CLASS_REFERENCE).getCallerClass());

and actually, I don't see it being used (outside of the tests for this testlibrary) and I don't think it will ever be used by actual tests that would use this framework, so I think we can safely remove it.

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should keep it for the cases when a user wants a more specific setup for his tests in the calling test class. For example, if he wants to define a new default warm-up for all of his tests (using setDefaultWarmup) or if he needs to combine different options for which there is no runXX method (helper classes, scenarios, and/or flags). In this case, he needs to create a TestFramework() builder and then call the different builder methods. We could argue that a single constructor is enough in which the test class needs to be specified. But I think most of the time, the user wants to test the calling class for which this additional constructor is quite useful.

Loading

Copy link
Member

@iignatev iignatev Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I think most of the time, the user wants to test the calling class for which this additional constructor is quite useful.

I concur that in most cases the users would want to use the caller as a testClass, yet we already have run and friends to cover that (most common) use-case, users who would need to construct fancier TestFramework, most probably, won't be doing it in their testClass, and if they would, the overhead of writing new TestFramework(getClass()) is just 10 keystrokes and is neglatible (given the assumption of a more complicated use case)

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is true. The entire test will probably look more complex in this case so it's probably reasonable to only provide the constructor with a class argument. However, if we are going to remove most of the (current) runXX() methods as discussed in a comment below, then keeping this parameterless constructor might be justified again? Anyhow, I'm fine with both. (Nit: when used from main() we would need to use MyTestClass.class instead of getClass())

Loading

* the entries of this list as a substring (partial match).
*/
public static final Set<String> JTREG_WHITELIST_FLAGS = new HashSet<>(
Arrays.asList(
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it doesn't seem to be a comprehensive list of flags that don't affect IR verification work, e.g. different GCs. I think it might be easy to go with the blacklist instead, and possibly warning people if there are any flags in test.*.opts.

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe first some general background about having a whitelist/blacklist in the first place. When a user writes an @IR rule, we do not want him to think about any possible VM flag combination (which he is not specifically using in his test, for example with addFlags) that could break his @IR rule and then requiring him to restrict the @IR rule with IR::applyIfXX() for these flags.

I agree that the whitelist is probably not complete and could be improved (but I think we should not whitelist different GCs as these affect the barrier code of the IR). About whitelist vs. blacklist, I had discussions about it with @TobiHartmann. We eventually decided to use a whitelist which seemed easier to maintain and is probably a lot shorter than a blacklist. In addition, if a new flag is introduced or removed, we most likely do not need to worry about updating the whitelist as the (currently) whitelisted flags will most likely remain unchanged. This first whitelist version is also just a best guess solution. We might miss on some other flags that will not affect the IR but we have the confidence that we do not run into surprising failures due to forgetting to blacklist a particular flag. But this is open for discussion of course. It would be interesting to hear how people think about it.

Loading

Copy link
Member

@iignatev iignatev Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll need to think about it a bit more, but my first impression after reading your comment is that we don't actually want to allow any flags, so we might better off just using @requies vm.flagless or reusing the underlying code to check that we don't have any flags that can "significantly" change behavior.

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our idea was to only restrict the IR verification done after the test VM terminates with the whitelist (or blacklist). But the framework should still run each test with any flag combination. This showed to be quite useful in Valhalla to find bugs.

Loading

* <li><p>If a helper class is not in the same file as the test class, make sure that JTreg compiles it by using
* {@literal @}compile in the JTreg header comment block.</li>
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need @compile to compile a helper class, 1st of all, you shouldn't use @compile when you want to compile a class in your test, you should use @build, 2nd, in this particular case, the class will be automatically built as part of your test b/c jtreg (or rather javac) will be able to statically detect it as a dependency of the code that calls runWithHelperClasses(Class, Class...)

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, thanks for clearing that up! I removed that comment and related ones.

Loading

run(walker.getCallerClass());
}

/**
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how much of the benefits all these different run.* bring. I fully understand the desire to simplify the most common use-case (i.e. no-arg run), but for the rest I'd assume the users will be capable of writing, for example, new TestFramework(testClass).addScenarios(scenarios).start(); instead of TestFramework .runWithScenarios(testClass, scenarios);

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are probably right. I added this builder approach later during the development of the framework. It might be better to keep the TestFramework interface simple(r). However, I will wait for others to comment on that before changing/removing all of these.

If we're gonna simplify it, I suggest to keep run() and maybe also runWithFlags() as (probably?) most common use-case.

Loading

Copy link
Member

@iignatev iignatev Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to keep run() and maybe also runWithFlags() as (probably?) most common use-case.

I guess you are right, run() and runWithFlags(String[]) will be most-commonly used ones.

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Will do these changes if others agree.

Loading

private List<Class<?>> helperClasses = null;
private List<Scenario> scenarios = null;
private final List<String> flags = new ArrayList<>();
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did you decide to eagerly create list for flags, but not for scenarios / helpersClasses?

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I adapted it to also create it lazily.

Loading

try {
start(null);
} catch (TestVMException e) {
System.err.println("\n" + e.getExceptionInfo());
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you shouldn't use "\n", as it might not be the right line-separator. you can either do:

Suggested change
System.err.println("\n" + e.getExceptionInfo());
System.err.println();
System.err.println(e.getExceptionInfo());

or

Suggested change
System.err.println("\n" + e.getExceptionInfo());
System.err.printf("%n%s%n", e.getExceptionInfo());

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will address all \n in a separate commit as there are a lot of them. Could I also just use System.lineSeparator() anywhere where I'm currently using \n like System.err.println(System.lineSeparator() + e.getExceptionInfo())? Might be easier when I'm using \n with a StringBuilder, for example.

Loading

Copy link
Member

@iignatev iignatev Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure you case use System.lineSeparator(), it's just a matter of personal choice/style

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I fixed them.

Loading

VERIFY_IR = hasIRAnnotations();
if (!VERIFY_IR) {
System.out.println("IR verification disabled due to test " + testClass + " not specifying any @IR annotations");
return;
}

VERIFY_IR = Platform.isDebugBuild() && !Platform.isInt() && !Platform.isComp();
if (!VERIFY_IR) {
System.out.println("IR verification disabled due to not running a debug build (required for PrintIdeal" +
"and PrintOptoAssembly), running with -Xint, or -Xcomp (use warm-up of 0 instead)");
return;
}
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd reorder them as "platform" checks are faster than hasIRAnnotations check and, I'd guess, will be a more common culprit to disable IR verification.

Loading

/**
* Disable IR verification completely in certain cases.
*/
private void maybeDisableIRVerificationCompletely() {
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
private void maybeDisableIRVerificationCompletely() {
private void disableIRVerificationIfNotFeasible() {

Loading

if (e instanceof IRViolationException) {
IRViolationException irException = (IRViolationException) e;
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
if (e instanceof IRViolationException) {
IRViolationException irException = (IRViolationException) e;
if (e instanceof IRViolationException irException) {

Loading

System.out.println((scenario != null ? "Scenario #" + scenario.getIndex() + " - " : "")
+ "Compilation(s) of failed matche(s):");
System.out.println(irException.getCompilations());
builder.append(errorMsg).append("\n").append(irException.getExceptionInfo()).append(e.getMessage());
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as builder.toString will be printed out to cout/cerr, you should use platform-specific line-separator instead of \n

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See earlier comment.

Loading

Copy link
Member

@iignatev iignatev left a comment

next portion of my comments, stay tuned for more to come :)

-- Igor

PS I must say that by some reason github's 'file changed' tab is unbelievably slow for me in this particular PR, so it's somewhat painful to even just find the place I want to associate my comment with (not to speak about actually using it to browse/read the code)

Loading

TestFormat.check(!scenarioIndices.contains(scenarioIndex),
"Cannot define two scenarios with the same index " + scenarioIndex);
scenarioIndices.add(scenarioIndex);
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can use Set::add to verify that the element isn't in a set:

Suggested change
TestFormat.check(!scenarioIndices.contains(scenarioIndex),
"Cannot define two scenarios with the same index " + scenarioIndex);
scenarioIndices.add(scenarioIndex);
TestFormat.check(scenarioIndices.add(scenarioIndex),
"Cannot define two scenarios with the same index " + scenarioIndex);

Loading

Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, shouldn't this check be done as part of addScenarios?

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I moved it to addScenarios.

Loading

/**
* Dedicated socket to send data from the flag and test VM back to the driver VM.
*/
class TestFrameworkSocket {
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please move it to a separate .java file?

Loading

/**
* Dedicated socket to send data from the flag and test VM back to the driver VM.
*/
class TestFrameworkSocket {
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it should implement AutoClosable, and then you try-catch-finally in TestFramework::start could be replaced by try-w-resource.

btw, I don't the fact that socket is a field of TestFramework with the lifetime bounded to start method.

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for AutoClosable. I wanted to avoid passing the socket object to all the different methods and thus tried to use the socket field like that. I agree that this not nice. Is delegating the socket object from TestFramework::start to the different methods the only solution or are there any other recommended ways?

Loading

if (exitCode != 0) {
System.err.println("--- OUTPUT TestFramework flag VM ---");
System.err.println(flagVMOutput);
throw new RuntimeException("\nTestFramework flag VM exited with " + exitCode);
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the reason for \n in the begging of this exception message?

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it's not necessary. Removed it.

Loading

Copy link
Member

@iignatev iignatev Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 . I noticed that you have leading \n in other exceptions, do you plan to remove it from there as well?

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes, I found the resulting format of the error message nicer with the additional new lines. For example the exception at L940:

JavaTest Message: Test threw exception: jdk.test.lib.hotspot.ir_framework.TestFormatException: 

Violations (1)
--------------
 - Cannot use explicit compile command annotations ...

vs. no new lines:

JavaTest Message: Test threw exception: jdk.test.lib.hotspot.ir_framework.TestFormatException: Violations (1)
--------------
 - Cannot use explicit compile command annotations ...

I will check the other exceptions again if they really need the new lines or not.

Loading


private void checkFlagVMExitCode(OutputAnalyzer oa) {
String flagVMOutput = oa.getOutput();
final int exitCode = oa.getExitValue();
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: there is no need for this final

Loading

Pattern pattern = Pattern.compile("compile_id='(\\d+)'.*" + Pattern.quote(testClass.getCanonicalName()) + " (\\S+)");
Matcher matcher = pattern.matcher(line);
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similarly to parseIREncoding, pattern here can be precomputed in IRMatcher::IRMatcher and stored into a final instance field.

Loading

Pattern pattern = Pattern.compile("compile_id='(\\d+)'");
Matcher matcher = pattern.matcher(line);
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again, precompiled pattern can be saved into a (in this case) static field and reused.

Loading

String line = lines[i].trim();
String[] splitComma = line.split(",");
if (splitComma.length < 2) {
throw new TestFrameworkException("Invalid IR match rule encoding");
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will it make sense to include the line-offender into the exception message here?

Loading

throw new TestFrameworkException("Invalid IR match rule encoding");
}
String testName = splitComma[0];
Integer[] irRulesIdx = new Integer[splitComma.length - 1];
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can actually use an array of int here, so there will be less wasted memory and no unboxing later on

Loading

*/
private void parseHotspotPidFile() {
Map<Integer, String> compileIdMap = new HashMap<>();
try (BufferedReader br = new BufferedReader(new FileReader(new File(System.getProperty("user.dir") + File.separator + hotspotPidFileName)))) {
Copy link
Member

@iignatev iignatev Apr 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more idiomatic/modern version would be

Suggested change
try (BufferedReader br = new BufferedReader(new FileReader(new File(System.getProperty("user.dir") + File.separator + hotspotPidFileName)))) {
try (BufferedReader br = Files.newBufferedReader(Paths.get(System.getProperty("user.dir"), hotspotPidFileName))) {

I actually not sure if you really need $user.dir, won't hotspot_pid file get generated in the current dir?

Loading

Copy link
Member Author

@chhagedorn chhagedorn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it should. Would this be equivalent to Paths.get("")? It seems to work like that.

Loading

Copy link
Member

@iignatev iignatev Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Paths.get(hotspotPidFileName) returns you a Path to hotspotPidFileName in cwd.

Loading

@chhagedorn
Copy link
Member Author

@chhagedorn chhagedorn commented May 4, 2021

As mentioned above, I moved the framework classes and tests and updated the packages. I also split the classes into subpackages to structure the code better. A summary of the package structure can be found in the updated README file in section 5.

I put this PR on hold as I'm away until the end of the month.

Loading

/**
* Selecting both the C1 and C2 compiler. This must be in sync with hotspot/share/compiler/compilerDefinitions.hpp.
*/
ANY(-2),
Copy link
Member

@TobiHartmann TobiHartmann May 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After the removal of AOT, this should be -1.

Loading

* <li><p>{@link DontCompile @DontCompile}: Prevents any compilation of the associated helper method.</li>
* </ul>
*/
ANY(-2),
Copy link
Member

@TobiHartmann TobiHartmann May 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After the removal of AOT, this should be -1.

Loading

@openjdk openjdk bot removed the ready label May 21, 2021
@chhagedorn
Copy link
Member Author

@chhagedorn chhagedorn commented May 31, 2021

I'm back to work and "reopen" this PR again.

@TobiHartmann Good point! I merged the master branch and fixed this to -1 together with some other test failures that emerged after the update.

Loading

@chhagedorn
Copy link
Member Author

@chhagedorn chhagedorn commented Jun 2, 2021

/contributor remove @TobiHartmann

Loading

@openjdk
Copy link

@openjdk openjdk bot commented Jun 2, 2021

@chhagedorn
Contributor Tobias Hartmann <thartmann@openjdk.org> successfully removed.

Loading

@chhagedorn
Copy link
Member Author

@chhagedorn chhagedorn commented Jun 2, 2021

/contributor add @chhagedorn

Loading

@chhagedorn
Copy link
Member Author

@chhagedorn chhagedorn commented Jun 2, 2021

/contributor add @TobiHartmann

Loading

@openjdk
Copy link

@openjdk openjdk bot commented Jun 2, 2021

@chhagedorn
Contributor Christian Hagedorn <chagedorn@openjdk.org> successfully added.

Loading

@openjdk
Copy link

@openjdk openjdk bot commented Jun 2, 2021

@chhagedorn
Contributor Tobias Hartmann <thartmann@openjdk.org> successfully added.

Loading

Copy link
Member

@iignatev iignatev left a comment

LGTM, modulo two nits in README file and the location of tests and examples: I'd prefer to place them into /test/hotspot/jtreg/testlibrary_tests/ir_framework and add explicit reference to that location in README file.

Loading

There are various ways how to set up and run a test within the `main()` method of a JTreg test. These are described and can be found in the [TestFramework](./TestFramework.java) class.

## 2. Features
The framework offers various annotations and flags to control how your test code should be invoked and being checked. This section gives an overview over all these features.
Copy link
Member

@iignatev iignatev Jun 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

redundant space at the beginning of the line

Loading

- `compiler.lib.ir_framework.shared`: These classes can be called from either the driver, flag, or test VM.

## 6. Summary
The initial design and feature set was kept simple and straight forward and serves well for small to medium sized tests. There are a lot of possibilities to further enhance the framework and make it more powerful. This can be tackled in additional RFEs. A few ideas can be found as subtasks of the [initial RFE](https://bugs.openjdk.java.net/browse/JDK-8254129) for this framework.
Copy link
Member

@iignatev iignatev Jun 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

redundant space:

Suggested change
The initial design and feature set was kept simple and straight forward and serves well for small to medium sized tests. There are a lot of possibilities to further enhance the framework and make it more powerful. This can be tackled in additional RFEs. A few ideas can be found as subtasks of the [initial RFE](https://bugs.openjdk.java.net/browse/JDK-8254129) for this framework.
The initial design and feature set was kept simple and straight forward and serves well for small to medium sized tests. There are a lot of possibilities to further enhance the framework and make it more powerful. This can be tackled in additional RFEs. A few ideas can be found as subtasks of the [initial RFE](https://bugs.openjdk.java.net/browse/JDK-8254129) for this framework.

Loading

@openjdk openjdk bot added the ready label Jun 3, 2021