New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accessing reflection pool #37

Closed
czyzby opened this Issue May 23, 2016 · 11 comments

Comments

Projects
None yet
2 participants
@czyzby

czyzby commented May 23, 2016

Usually most platforms include a way of checking which classes are available for the application, which allows to create tools like annotated classes scanners. Regular JVM allow to browse classpath files, while on Android you can use internal ApplicationInfo API to access the whole class pool. Is there a similar functionality in JTransc? Is the generated reflection pool somehow available through JTransc API? I can imagine that class objects might be simply assigned directly to each class and not stored globally, but it would be nice if you could access this meta-date en masse.

@soywiz

This comment has been minimized.

Show comment
Hide comment
@soywiz

soywiz May 23, 2016

Member

Which is the standard way of doing this?

I have seen this:
http://docs.oracle.com/javase/6/docs/api/java/lang/instrument/Instrumentation.html#getInitiatedClasses(java.lang.ClassLoader)

I can implement a custom way for doing this in JTransc. Or provide a standard way if there is any.

I'm not embedding .class files, so you cannot iterate folders. Though I could fake those files when listing files in the filesystem.

Member

soywiz commented May 23, 2016

Which is the standard way of doing this?

I have seen this:
http://docs.oracle.com/javase/6/docs/api/java/lang/instrument/Instrumentation.html#getInitiatedClasses(java.lang.ClassLoader)

I can implement a custom way for doing this in JTransc. Or provide a standard way if there is any.

I'm not embedding .class files, so you cannot iterate folders. Though I could fake those files when listing files in the filesystem.

@czyzby

This comment has been minimized.

Show comment
Hide comment
@czyzby

czyzby May 23, 2016

ClassScanner API provides ways of reading classpath files - from what I can see, most class scanners use it to access the classes (especially since this allows to analyze sources without actually loading the classes themselves). Files faking approach certainly isn't the way to go. I don't think there's a standard ways of doing this on most JVMs, so I wouldn't mind JTransc-specific API to access all classes - if that won't lead to too much runtime overhead, of course. You could make it optional (new compiler flag).

This would certainly help dependency injection frameworks with automatic component scan.

czyzby commented May 23, 2016

ClassScanner API provides ways of reading classpath files - from what I can see, most class scanners use it to access the classes (especially since this allows to analyze sources without actually loading the classes themselves). Files faking approach certainly isn't the way to go. I don't think there's a standard ways of doing this on most JVMs, so I wouldn't mind JTransc-specific API to access all classes - if that won't lead to too much runtime overhead, of course. You could make it optional (new compiler flag).

This would certainly help dependency injection frameworks with automatic component scan.

@soywiz

This comment has been minimized.

Show comment
Hide comment
@soywiz

soywiz May 23, 2016

Member

That should do the trick.
But it would be great if the body of JTranscSystem.getAllClasses() (bf1349b#diff-a657dfe14de91475dd308a5bfd97fec8R206) is implemented on JVM.
But I have to find a relieable method that doesn't involve including a project with lots of classes.

Member

soywiz commented May 23, 2016

That should do the trick.
But it would be great if the body of JTranscSystem.getAllClasses() (bf1349b#diff-a657dfe14de91475dd308a5bfd97fec8R206) is implemented on JVM.
But I have to find a relieable method that doesn't involve including a project with lots of classes.

@czyzby

This comment has been minimized.

Show comment
Hide comment
@czyzby

czyzby May 23, 2016

I implemented something similar. It would load the whole classpath though, so you can forget about load-on-demand class loading. (Then again, this is exactly the point in this particular case.) You'd have to remove the annotation scanning code though and include every single class in the result.

It uses some LibGDX-specific APIs, but once you get rid of LibGDX collections and utilities, it's pretty straightforward to convert.

czyzby commented May 23, 2016

I implemented something similar. It would load the whole classpath though, so you can forget about load-on-demand class loading. (Then again, this is exactly the point in this particular case.) You'd have to remove the annotation scanning code though and include every single class in the result.

It uses some LibGDX-specific APIs, but once you get rid of LibGDX collections and utilities, it's pretty straightforward to convert.

@czyzby

This comment has been minimized.

Show comment
Hide comment
@czyzby

czyzby May 23, 2016

By the way, I still think you should make it optional to fill this array - add a new property to JTranscSystem (false by default, since it's a pretty specific functionality) or a compiler flag, so the users could limit the runtime meta-data.

czyzby commented May 23, 2016

By the way, I still think you should make it optional to fill this array - add a new property to JTranscSystem (false by default, since it's a pretty specific functionality) or a compiler flag, so the users could limit the runtime meta-data.

@czyzby

This comment has been minimized.

Show comment
Hide comment
@czyzby

czyzby May 23, 2016

If you haven't started messing around with the code, I can refactor it and create a pull request. Now that I look at it, it seems pretty unclean.

czyzby commented May 23, 2016

If you haven't started messing around with the code, I can refactor it and create a pull request. Now that I look at it, it seems pretty unclean.

@soywiz

This comment has been minimized.

Show comment
Hide comment
@soywiz

soywiz May 23, 2016

Member

@czyzby Sure. Not started yed :) do a PR 👍
(Now I'm with the black screen issue)

Member

soywiz commented May 23, 2016

@czyzby Sure. Not started yed :) do a PR 👍
(Now I'm with the black screen issue)

@czyzby

This comment has been minimized.

Show comment
Hide comment
@czyzby

czyzby May 23, 2016

OK, give me a day or so.

czyzby commented May 23, 2016

OK, give me a day or so.

@czyzby

This comment has been minimized.

Show comment
Hide comment
@czyzby

czyzby May 24, 2016

I converted my class scanner, but I'm not sure if it should be posted. ClassLoader approach does not seem to give access to every class out there, it just simply allows to scan the unique classes of the current application, but does not list anything from the system library, for example. Anyway, here's a huge snippet:


import java.io.File;
import java.io.IOException;
import java.net.URISyntaxException;
import java.net.URL;
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.HashSet;
import java.util.LinkedList;
import java.util.List;
import java.util.Queue;
import java.util.Set;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;

public class ClasspathScanner {
    private static final String CLASS_FILE_EXTENSION = ".class";
    private static final String JAR_FILE_EXTENSION = ".jar";

    public String[] getAllClasses() {
        final ClassLoader classLoader = getClass().getClassLoader();
        try {
            final Enumeration<URL> resources = classLoader.getResources("");
            final Queue<DepthFile> filesWithDepthsToProcess = new LinkedList<DepthFile>();
            while (resources.hasMoreElements()) {
                try {
                    filesWithDepthsToProcess.add(new DepthFile(0, toFile(resources.nextElement())));
                } catch (final Exception uriSyntaxException) {
                    // Will throw an exception for non-hierarchical files. Ignored.
                }
            }
            final Set<String> classNames = new HashSet<String>();
            if (filesWithDepthsToProcess.isEmpty()) {
                extractFromJar(classLoader, classNames);
            }
            extractFromBinaries(filesWithDepthsToProcess, classNames);
            return classNames.toArray(new String[classNames.size()]);
        } catch (final Exception exception) {
            throw new RuntimeException("Unable to scan classpath.", exception);
        }
    }

    private static void extractFromBinaries(final Queue<DepthFile> filesWithDepthsToProcess,
            final Set<String> classNames) throws Exception {
        while (!filesWithDepthsToProcess.isEmpty()) {
            final DepthFile classPathFileWithDepth = filesWithDepthsToProcess.poll();
            final File classPathFile = classPathFileWithDepth.file;
            final int depth = classPathFileWithDepth.depth;
            if (classPathFile.isDirectory()) {
                addAllChildren(filesWithDepthsToProcess, classPathFile, depth);
            } else {
                final String className = getBinaryClassName(classPathFile, depth);
                if (isNotPackageInfo(className)) {
                    classNames.add(className);
                }
            }
        }
    }

    private static boolean isNotPackageInfo(final String className) {
        return className.indexOf('-') < 0;
    }

    private static File toFile(final URL url) throws URISyntaxException {
        return new File(url.toURI()).getAbsoluteFile();
    }

    private static void addAllChildren(final Queue<DepthFile> rootFiles, final File classPathFile, int depth) {
        depth++;
        for (final File file : classPathFile.listFiles()) {
            if (file.isDirectory() || file.getName().endsWith(CLASS_FILE_EXTENSION)) {
                rootFiles.add(new DepthFile(depth, file));
            }
        }
    }

    private static String getBinaryClassName(final File classPathFile, final int depth) {
        final String[] classFolders = classPathFile.getPath().split(File.separator);
        final StringBuilder builder = new StringBuilder();
        for (int folderIndex = classFolders.length - depth; folderIndex < classFolders.length - 1; folderIndex++) {
            if (builder.length() > 0) {
                builder.append('.');
            }
            builder.append(classFolders[folderIndex]);
        }
        final String classFileName = classFolders[classFolders.length - 1];
        builder.append('.').append(classFileName.substring(0, classFileName.length() - CLASS_FILE_EXTENSION.length()));
        return builder.toString();
    }

    private static void extractFromJar(final ClassLoader classLoader, final Set<String> classNames) throws Exception {
        final List<JarFile> filesToProcess = getJarFilesToProcess();
        for (final JarFile jarFile : filesToProcess) {
            final Enumeration<JarEntry> entries = jarFile.entries();
            while (entries.hasMoreElements()) {
                final JarEntry entry = entries.nextElement();
                processEntry(entry, classNames);
            }
        }
    }

    private static List<JarFile> getJarFilesToProcess() throws URISyntaxException, IOException {
        final List<JarFile> filesToProcess = new ArrayList<JarFile>();
        final File jarDirectory = new File(ClassLoader.getSystemClassLoader().getResource(".").toURI());
        for (final File file : jarDirectory.listFiles()) {
            if (file.getName().endsWith(JAR_FILE_EXTENSION)) {
                filesToProcess.add(new JarFile(file));
            }
        }
        return filesToProcess;
    }

    private static void processEntry(final JarEntry entry, final Set<String> classNames) throws Exception {
        if (!entry.isDirectory()) {
            final String entryName = entry.getName();
            if (entryName.endsWith(CLASS_FILE_EXTENSION) && !isNotPackageInfo(entryName)) {
                classNames.add(jarEntryToClassName(entryName));
            }
        }
    }

    private static String jarEntryToClassName(final String entryName) {
        return entryName.substring(0, entryName.length() - CLASS_FILE_EXTENSION.length()).replace(File.separatorChar,
                '.');
    }

    private static class DepthFile {
        private final int depth;
        private final File file;

        public DepthFile(final int depth, final File file) {
            this.depth = depth;
            this.file = file;
        }
    }
}

You can include it, of course, but I didn't submit a PR, as the functionality is obviously flawed. You can, however, try expanding it with processing entries from System.getProperty("java.class.path") or the Instrumentation API you mentioned.

czyzby commented May 24, 2016

I converted my class scanner, but I'm not sure if it should be posted. ClassLoader approach does not seem to give access to every class out there, it just simply allows to scan the unique classes of the current application, but does not list anything from the system library, for example. Anyway, here's a huge snippet:


import java.io.File;
import java.io.IOException;
import java.net.URISyntaxException;
import java.net.URL;
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.HashSet;
import java.util.LinkedList;
import java.util.List;
import java.util.Queue;
import java.util.Set;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;

public class ClasspathScanner {
    private static final String CLASS_FILE_EXTENSION = ".class";
    private static final String JAR_FILE_EXTENSION = ".jar";

    public String[] getAllClasses() {
        final ClassLoader classLoader = getClass().getClassLoader();
        try {
            final Enumeration<URL> resources = classLoader.getResources("");
            final Queue<DepthFile> filesWithDepthsToProcess = new LinkedList<DepthFile>();
            while (resources.hasMoreElements()) {
                try {
                    filesWithDepthsToProcess.add(new DepthFile(0, toFile(resources.nextElement())));
                } catch (final Exception uriSyntaxException) {
                    // Will throw an exception for non-hierarchical files. Ignored.
                }
            }
            final Set<String> classNames = new HashSet<String>();
            if (filesWithDepthsToProcess.isEmpty()) {
                extractFromJar(classLoader, classNames);
            }
            extractFromBinaries(filesWithDepthsToProcess, classNames);
            return classNames.toArray(new String[classNames.size()]);
        } catch (final Exception exception) {
            throw new RuntimeException("Unable to scan classpath.", exception);
        }
    }

    private static void extractFromBinaries(final Queue<DepthFile> filesWithDepthsToProcess,
            final Set<String> classNames) throws Exception {
        while (!filesWithDepthsToProcess.isEmpty()) {
            final DepthFile classPathFileWithDepth = filesWithDepthsToProcess.poll();
            final File classPathFile = classPathFileWithDepth.file;
            final int depth = classPathFileWithDepth.depth;
            if (classPathFile.isDirectory()) {
                addAllChildren(filesWithDepthsToProcess, classPathFile, depth);
            } else {
                final String className = getBinaryClassName(classPathFile, depth);
                if (isNotPackageInfo(className)) {
                    classNames.add(className);
                }
            }
        }
    }

    private static boolean isNotPackageInfo(final String className) {
        return className.indexOf('-') < 0;
    }

    private static File toFile(final URL url) throws URISyntaxException {
        return new File(url.toURI()).getAbsoluteFile();
    }

    private static void addAllChildren(final Queue<DepthFile> rootFiles, final File classPathFile, int depth) {
        depth++;
        for (final File file : classPathFile.listFiles()) {
            if (file.isDirectory() || file.getName().endsWith(CLASS_FILE_EXTENSION)) {
                rootFiles.add(new DepthFile(depth, file));
            }
        }
    }

    private static String getBinaryClassName(final File classPathFile, final int depth) {
        final String[] classFolders = classPathFile.getPath().split(File.separator);
        final StringBuilder builder = new StringBuilder();
        for (int folderIndex = classFolders.length - depth; folderIndex < classFolders.length - 1; folderIndex++) {
            if (builder.length() > 0) {
                builder.append('.');
            }
            builder.append(classFolders[folderIndex]);
        }
        final String classFileName = classFolders[classFolders.length - 1];
        builder.append('.').append(classFileName.substring(0, classFileName.length() - CLASS_FILE_EXTENSION.length()));
        return builder.toString();
    }

    private static void extractFromJar(final ClassLoader classLoader, final Set<String> classNames) throws Exception {
        final List<JarFile> filesToProcess = getJarFilesToProcess();
        for (final JarFile jarFile : filesToProcess) {
            final Enumeration<JarEntry> entries = jarFile.entries();
            while (entries.hasMoreElements()) {
                final JarEntry entry = entries.nextElement();
                processEntry(entry, classNames);
            }
        }
    }

    private static List<JarFile> getJarFilesToProcess() throws URISyntaxException, IOException {
        final List<JarFile> filesToProcess = new ArrayList<JarFile>();
        final File jarDirectory = new File(ClassLoader.getSystemClassLoader().getResource(".").toURI());
        for (final File file : jarDirectory.listFiles()) {
            if (file.getName().endsWith(JAR_FILE_EXTENSION)) {
                filesToProcess.add(new JarFile(file));
            }
        }
        return filesToProcess;
    }

    private static void processEntry(final JarEntry entry, final Set<String> classNames) throws Exception {
        if (!entry.isDirectory()) {
            final String entryName = entry.getName();
            if (entryName.endsWith(CLASS_FILE_EXTENSION) && !isNotPackageInfo(entryName)) {
                classNames.add(jarEntryToClassName(entryName));
            }
        }
    }

    private static String jarEntryToClassName(final String entryName) {
        return entryName.substring(0, entryName.length() - CLASS_FILE_EXTENSION.length()).replace(File.separatorChar,
                '.');
    }

    private static class DepthFile {
        private final int depth;
        private final File file;

        public DepthFile(final int depth, final File file) {
            this.depth = depth;
            this.file = file;
        }
    }
}

You can include it, of course, but I didn't submit a PR, as the functionality is obviously flawed. You can, however, try expanding it with processing entries from System.getProperty("java.class.path") or the Instrumentation API you mentioned.

@soywiz soywiz closed this in e4453ec May 25, 2016

@soywiz

This comment has been minimized.

Show comment
Hide comment
@soywiz

soywiz May 25, 2016

Member

I will add it anyway. But I'm going to move the method to its own class (probably JTranscReflection), so this code is not included unless you use that method.

Member

soywiz commented May 25, 2016

I will add it anyway. But I'm going to move the method to its own class (probably JTranscReflection), so this code is not included unless you use that method.

@czyzby

This comment has been minimized.

Show comment
Hide comment
@czyzby

czyzby May 26, 2016

That's great you found it useful, but you should be aware that it won't match the exact "native" Haxe implementation. We would have to scan third party libraries and system jars to do that.

czyzby commented May 26, 2016

That's great you found it useful, but you should be aware that it won't match the exact "native" Haxe implementation. We would have to scan third party libraries and system jars to do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment