Skip to content

Conversation

@ascopes
Copy link
Owner

@ascopes ascopes commented May 20, 2022

This rewrites the path file management system from scratch to use a mechanism that is closer to how OpenJDK Javac works under the hood.

The implementation works by storing a map of locations to container groups in the file manager.

The file manager (part of the JSR-199 compiler API in the JDK) has a one-to-many relationship with container groups. Each container group has a one-to-many relationship with containers. Each container has a 1-to-1 relationship with some form of Path-like object.

Each container group can supply a classloader, and provides some form of access to "containers". The container group implementations fall into three categories:

  • Package-oriented - these contain a list of containers, designed for regular packages.
  • Module-oriented - these contain a mapping of module locations to package-oriented container groups. The purpose of these is to store locations that contain JPMS modules. Module-oriented container groups cannot contain files outside of the nested package-oriented groups.
  • Output-oriented - these can contain both package-oriented containers and a mapping of modules, just like module-oriented. This implementation is designed to house output locations such as SOURCE_OUTPUT and CLASS_OUTPUT which may contain a single package tree or a collection of JPMS modules.

Each container group holds containers. Containers are wrappers around single path-like objects, and provide an interface for consistent read, write, and list operations on the file tree associated with the path. There are currently three implementations of container:

  • Directory container - this holds a reference to a Path that is on some file system.
  • RamPath container - this extends the Directory container by also holding a reference to the RamPath associated with the Path. This enables holding a strong reference to RamPath objects to prevent premature garbage collection. The mechanism is needed since RamPath objects hold an in-memory JIMFS FileSystem internally that gets closed once the enclosing RamPath is garbage collected.
  • JAR container - this holds a ZipFileSystem that has Multi-Release configuration enabled. The purpose here is to allow reading a JAR or WAR of files. JAR containers do not support opening files for output, and are "mounted" as read-only. A JarFileSystem is not used for this since the OpenJDK JAR filesystem provider implementation can not open more than one copy of the same ZIP-like archive at once, which prevents tests using JCT from running safely in parallel. Conversely the ZipFileSystem provider that JarFileSystem extends does not appear to have this limitation. This entire mechanism replaces the need to create temporary symbolic links in the default system FileSystem to work around the JarFileSystem constraints explained here.

A side effect of supporting Multi-Release in the MANIFEST.MF is that we have to hold the target compiler version number string, which effectively prevents this entire mechanism from being constructed until this release version is known. This means that the existing process for adding new paths to a Compiler implementation will have to store all paths and locations in a separate structure, only registering them fully once a FileManager is created in the SimpleCompilationFactory.

The majority of the "old" paths implementation will be removed entirely as part of this PR.

Repository owner deleted a comment from github-actions bot May 20, 2022
@github-actions
Copy link

github-actions bot commented May 21, 2022

Unit Test Results

     608 files   -      32       608 suites   - 32   19m 4s ⏱️ + 7m 10s
  7 192 tests  -    910    7 127 ✔️  -    677    65 💤  - 233  0 ±0 
13 088 runs   - 1 632  12 944 ✔️  - 1 232  144 💤  - 400  0 ±0 

Results for commit 3240e7b. ± Comparison against base commit 8d0d8f8.

♻️ This comment has been updated with latest results.

@ascopes ascopes force-pushed the task/reimplement-paths branch from 7c05e1c to 4aaf5be Compare May 21, 2022 15:42
@lgtm-com
Copy link

lgtm-com bot commented May 21, 2022

This pull request fixes 1 alert when merging 4aaf5be into 8d0d8f8 - view on LGTM.com

fixed alerts:

  • 1 for Container contents are never accessed

@codecov-commenter
Copy link

codecov-commenter commented May 22, 2022

Codecov Report

Merging #33 (ee28ada) into main (8d0d8f8) will decrease coverage by 12.10%.
The diff coverage is 47.62%.

@@             Coverage Diff             @@
##             main      #33       +/-   ##
===========================================
- Coverage   70.61%   58.51%   -12.11%     
===========================================
  Files          42       63       +21     
  Lines        1746     2114      +368     
  Branches      108      155       +47     
===========================================
+ Hits         1233     1237        +4     
- Misses        474      817      +343     
- Partials       39       60       +21     
Impacted Files Coverage Δ
...hub/ascopes/jct/assertions/AbstractEnumAssert.java 0.00% <0.00%> (ø)
...copes/jct/assertions/AbstractFileObjectAssert.java 0.00% <0.00%> (ø)
...ithub/ascopes/jct/assertions/DiagnosticAssert.java 0.00% <0.00%> (ø)
...b/ascopes/jct/assertions/DiagnosticKindAssert.java 0.00% <0.00%> (ø)
...copes/jct/assertions/DiagnosticRepresentation.java 0.00% <0.00%> (ø)
...thub/ascopes/jct/assertions/FileManagerAssert.java 0.00% <0.00%> (ø)
...ithub/ascopes/jct/assertions/FileObjectAssert.java 0.00% <0.00%> (ø)
...b/ascopes/jct/assertions/JavaFileObjectAssert.java 0.00% <0.00%> (ø)
...copes/jct/assertions/JavaFileObjectKindAssert.java 0.00% <0.00%> (ø)
...ithub/ascopes/jct/assertions/StackTraceAssert.java 0.00% <0.00%> (ø)
... and 60 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d0d8f8...ee28ada. Read the comment docs.

@lgtm-com
Copy link

lgtm-com bot commented May 22, 2022

This pull request fixes 1 alert when merging f0c1ba0 into 8d0d8f8 - view on LGTM.com

fixed alerts:

  • 1 for Container contents are never accessed

@lgtm-com
Copy link

lgtm-com bot commented May 23, 2022

This pull request fixes 1 alert when merging 025cc76 into 8d0d8f8 - view on LGTM.com

fixed alerts:

  • 1 for Container contents are never accessed

@ascopes ascopes force-pushed the task/reimplement-paths branch from 025cc76 to 0d868aa Compare May 23, 2022 10:30
@lgtm-com
Copy link

lgtm-com bot commented May 23, 2022

This pull request fixes 1 alert when merging 0d868aa into 8d0d8f8 - view on LGTM.com

fixed alerts:

  • 1 for Container contents are never accessed

@lgtm-com
Copy link

lgtm-com bot commented May 23, 2022

This pull request fixes 1 alert when merging 004f479 into 8d0d8f8 - view on LGTM.com

fixed alerts:

  • 1 for Container contents are never accessed

@lgtm-com
Copy link

lgtm-com bot commented May 28, 2022

This pull request fixes 1 alert when merging 74b77b0 into 8d0d8f8 - view on LGTM.com

fixed alerts:

  • 1 for Container contents are never accessed

@lgtm-com
Copy link

lgtm-com bot commented May 28, 2022

This pull request fixes 1 alert when merging b06d4fd into 8d0d8f8 - view on LGTM.com

fixed alerts:

  • 1 for Container contents are never accessed

It appears the file management for modules in ECJ is totally bypassing the mechanism
we implement to read files and query the compilation unit file system. Instead, it is
using org.eclipse.jdt.internal.compiler.batch.FileSystem#initializeKnownFileNames and
using java.io.File, which will not work with non-default file systems. As for why this
works without modules, I have no idea.
@lgtm-com
Copy link

lgtm-com bot commented May 28, 2022

This pull request fixes 1 alert when merging 2b7e79e into 8d0d8f8 - view on LGTM.com

fixed alerts:

  • 1 for Container contents are never accessed

@lgtm-com
Copy link

lgtm-com bot commented May 29, 2022

This pull request fixes 1 alert when merging 46c819c into 8d0d8f8 - view on LGTM.com

fixed alerts:

  • 1 for Container contents are never accessed

@ascopes ascopes force-pushed the task/reimplement-paths branch from 967c7a2 to 367670d Compare May 29, 2022 17:32
@ascopes ascopes force-pushed the task/reimplement-paths branch from 367670d to 69941a2 Compare May 29, 2022 18:01
export M2_HOME="${HOME:-$HOMEDIR}/.m2"
fi

workspace_dir="$(realpath $(dirname ${BASH_SOURCE[0]:-$0})/..)"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SC2046: Quote this to prevent word splitting.

(at-me in a reply with help or ignore)


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

-w "${container_workspace_dir}" \
--rm \
"${image}" \
${command} \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SC2086: Double quote to prevent globbing and word splitting.

(at-me in a reply with help or ignore)


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

command="${default_command}"

while getopts ":c:v:h" opt; do
case "${opt}" in
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SC2220: Invalid flags are not handled. Add a *) case.

(at-me in a reply with help or ignore)


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

export M2_HOME="${HOME:-$HOMEDIR}/.m2"
fi

workspace_dir="$(realpath $(dirname ${BASH_SOURCE[0]:-$0})/..)"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SC2086: Double quote to prevent globbing and word splitting.

(at-me in a reply with help or ignore)


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]


mkdir target 2>&1 || true

for version in $(seq ${first_version} ${last_version}); do
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SC2086: Double quote to prevent globbing and word splitting.

(at-me in a reply with help or ignore)


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

"${image}" \
${command} \
2>&1 |
while read line; do
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SC2162: read without -r will mangle backslashes.

(at-me in a reply with help or ignore)


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]


mkdir target 2>&1 || true

for version in $(seq ${first_version} ${last_version}); do
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SC2086: Double quote to prevent globbing and word splitting.

(at-me in a reply with help or ignore)


Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

@ascopes ascopes force-pushed the task/reimplement-paths branch from 4d59370 to cf3769a Compare May 30, 2022 12:11
@lgtm-com
Copy link

lgtm-com bot commented May 30, 2022

This pull request introduces 1 alert and fixes 1 when merging ee28ada into 8d0d8f8 - view on LGTM.com

new alerts:

  • 1 for Type mismatch on container access

fixed alerts:

  • 1 for Container contents are never accessed

Fix Windows runner issue that was propagating through assertj
@ascopes ascopes force-pushed the task/reimplement-paths branch from ee28ada to 3240e7b Compare May 30, 2022 12:23
@ascopes ascopes marked this pull request as ready for review May 30, 2022 12:30
@ascopes ascopes merged commit a32df18 into main May 30, 2022
@ascopes ascopes deleted the task/reimplement-paths branch May 30, 2022 12:31
@lgtm-com
Copy link

lgtm-com bot commented May 30, 2022

This pull request introduces 1 alert and fixes 1 when merging 3240e7b into 8d0d8f8 - view on LGTM.com

new alerts:

  • 1 for Type mismatch on container access

fixed alerts:

  • 1 for Container contents are never accessed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants