Validation not working when project is connected to Jazz source control

[ahojuk]

Validation is not working when project is connected to IBM Jazz source control.
Validation error is displayed after modifying robotframework files.
Validation error ruins RED's validation functionality so many RED's helpful capabilities are also not working.

Eclipse error log:

eclipse.buildId=4.7.3.M20180301-0715
java.version=1.8.0_102
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments: -product org.eclipse.epp.package.jee.product
Command-line arguments: -data file:/home/username/eclipse-workspace/ -os linux -ws gtk -arch x86_64 -product org.eclipse.epp.package.jee.product

org.eclipse.core.jobs
Error
Tue Apr 03 14:18:42 EEST 2018
An internal error occurred during: "Validating".

java.lang.IllegalArgumentException: URI scheme is not "file"
at java.io.File.(File.java:421)
at org.robotframework.ide.eclipse.main.plugin.model.RobotProject.lambda$1(RobotProject.java:233)
at org.robotframework.ide.eclipse.main.plugin.model.RobotProject.lambda$0(RobotProject.java:139)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
at com.google.common.collect.CollectSpliterators$1FlatMapSpliterator.lambda$forEachRemaining$1(CollectSpliterators.java:246)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at com.google.common.collect.CollectSpliterators$1FlatMapSpliterator.forEachRemaining(CollectSpliterators.java:246)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at org.robotframework.ide.eclipse.main.plugin.model.RobotProject.getStandardLibraries(RobotProject.java:138)
at org.robotframework.ide.eclipse.main.plugin.model.RobotProject.getLibrarySpecificationsStream(RobotProject.java:194)
at org.robotframework.ide.eclipse.main.plugin.model.RobotProject.getLibrarySpecifications(RobotProject.java:207)
at org.robotframework.ide.eclipse.main.plugin.project.build.validation.ValidationContext.collectLibraries(ValidationContext.java:102)
at org.robotframework.ide.eclipse.main.plugin.project.build.validation.ValidationContext.(ValidationContext.java:85)
at org.robotframework.ide.eclipse.main.plugin.project.build.RobotArtifactsValidator.runValidation(RobotArtifactsValidator.java:161)
at org.robotframework.ide.eclipse.main.plugin.project.build.RobotArtifactsValidator.access$3(RobotArtifactsValidator.java:148)
at org.robotframework.ide.eclipse.main.plugin.project.build.RobotArtifactsValidator$3.runInWorkspace(RobotArtifactsValidator.java:133)
at org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:39)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:56)

Assumption:
This error happens because RED is expecting URI Scheme to be "file:".
In my case IBM Jazz source control is used and URI scheme is returned as "sourcecontrol:".

My project's local folder structure:
/home/username/eclipse-workspace/project-name/red.xml

Similar URI scheme issue is also handled here:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=366652

[michalanglart]

Hi,

interesting case; we'll have a look. At first sight it seems that there are couple of places which can be broken by using scheme assumption.

Hopefully this can be fixed, but we need to investigate.

[bastidest]

Hi,

I did some debugging and it seems that the plugin expected that the URI scheme always is 'file:'.

IResource.getLocationURI() should be replaced byIResource.getLocation().toFile().toURI() to first get a handle to the file and the convert the file handle (insted of the source control handle) to a URI.

I applied this fix and everything seems to be working just fine and the tests are passing (https://github.com/bastidest/RED/commit/561192fb4b90a468d164b2f70ccf8b6ae2e6565b).

Are there any other dependencies I am not aware of? Do you suggest moving this code in a Util class?

Thank you

[ahojuk]

Great! I will test it when it's released.

[michalanglart]

@bastidest This looks alright at first sight. We would need to test how does it behave with some remotely linked filesystems (e.g. via Remote System Explorer) as some of our clients were using it in daily work.

[bastidest]

@michalanglart Sure, do you know how to test this / are there tests available? I don't know the framework very well (I just read 'if(foo.equals("Remote")' in some places).

[michalanglart]

if(foo.equals("Remote") has nothing to do with that. Those places are probably used to somehow handle RobotFramework standard library called Remote.

I was refering to eclipse plugin called Remote System Explorer. This is a plugin which allows user to see some remote filesystems like if it would be part of their workspace/projects. In short you're seeing in navigator the files which are not part of local filesystem but instead they are remotely connected for example via SSH. Those files however have different URI schemes than file:// so I wonder how it would work with those changes.

As far as I remember we do not have automatic tests for RSE integration since we've only had few minor issues raised for that and on the other hand it would require us to set up some environment in which this integration could be tested.

[bastidest]

I tried out working with the Remote System Explorer and created a project from a remote destination. It does not seem to be working both before and after my changes. Can anyone provide or test a concrete example/use case of a RED Project in conjunction with the RSE?

When I configure a RSE project folder as RED Project I get a NPE on this line:

RED/src/Eclipse-IDE/org.robotframework.ide.eclipse.main.plugin/src/org/robotframework/ide/eclipse/main/plugin/project/build/libs/StandardLibraryLibdocGenerator.java

Line 29 in bc91515

runtimeEnvironment.createLibdoc(libName, targetSpecFile.getLocation().toFile(), format);

I think getLocation tries to find a IResource inside the project folder, but it does not exist locally.

I moved the changes I made into a helper class and applied the scheme transformation only if the scheme is sourcecontrol.

The changes can be found here: bastidest/RED@58d031f50a152ae86e7a6b11928369b6197eff1f

Let me know if I should create a pull request.

[michalanglart]

I think getLocation tries to find a IResource inside the project folder, but it does not exist locally.

This is precisely what I was afraid of (NPE when projects are outside the local file system) when looking at proposal to replace all getLocationURI with getLocation().toFile().toURI().

This may be true that currently RED might again not work with remotely connected file systems (as I said we do not test it), however in the past we were making some changes in order to fix some problems for users working with such remote systems. Overall it would be the best to:

• make it again possible to work with remote file systems,
• fix the problem in this issue

together.

[michalanglart]

This problem can be reproduced without using IBM Jazz Source Control. It is sufficient to use any other plugin which is able to provide some local but non-file:-based file system. I was able to reproduce this exception using: https://github.com/sirnewton01/eclipse-filesystem-example (which I found when looking at similar issue in CDT: https://bugs.eclipse.org/bugs/show_bug.cgi?id=378882 ).

[bastidest]

Would you be comfortable with applying this workaround for IBM Jazz Source Control and opening another more general Issue for non-file:-based file systems?

public static URI getUri(IResource res) {
    final URI locationUri = res.getLocationURI();
    if(locationUri.getScheme().equals("sourcecontrol")) {
        return res.getLocation().toFile().toURI();
    } else {
        return locationUri;
    }
}

The Plugin would be perfectly functional for my use-case because the sourcecontrol: URI can be resolved to a local file: URI. The other use cases should behave just like before. For now I am stuck with copying the project files from jazz into a temporary local folder, making changes, and copying it back.

[michalanglart]

Would you be comfortable with applying this workaround for IBM Jazz Source Control and opening another more general Issue for non-file:-based file systems?

I think we don't need another issue. We'll just fix this in more general case. I'm looking into it and it seems we can fix that for next release.

As to your workaround proposed in https://github.com/bastidest/RED/commit/58d031f50a152ae86e7a6b11928369b6197eff1f : there are 2 problems with it:

• it does not solve the problem generally
• there are places which seems to work fine when getLocationURI is used and workaround is not needed - for example the debugger works properly despite the fact that non-file uris are used so changes in RobotBreakpoints and EclipseElementsLocator are not needed.

From my investigation it seems that there are 2 possible fixes:

public static URI getUri(IResource res) {
    if(res.getLocation() != null) {
        return res.getLocation().toFile().toURI();
    } else {
        return res.getLocationURI();
    }
}

or the second one:

public static URI getUri(IResource res) {
        final URI uri = res.getLocationURI();
        try {
            final IFileSystem fileSystem = EFS.getFileSystem(uri .getScheme());
            final IFileStore fileStore = fileSystem.getStore(uri);
            if (fileStore == null) {
                return uri;
            }
            final File localFile = fileStore.toLocalFile(EFS.NONE, new NullProgressMonitor()));
            return localFile == null ? uri : localFile.toURI();

        } catch (final CoreException e) {
            return uri;
        }
}

Both seem to work with the latter being more clear and according to javadocs for methods used it does precisely what we want to achieve. The first one seem to work too but I'm not yet sure if it is equivalent to second.

For now I am stuck with copying the project files from jazz into a temporary local folder, making changes, and copying it back.

Ugh, this sounds awfully bad. We'll make it work.

[ahojuk]

I tested this fix today. Looks like validation is now working ok with IBM Jazz source control.
Thanks a lot!

Tested in:
Ubuntu 16.04.4 LTS
Robot Framework 3.1a2
Eclipse Oxygen.3 Release (4.7.3)

• RED - Robot Editor v0.8.8
• IBM Rational Team Concert 3.1.1200.v20151110_0208

[michalanglart]

Great to hear that.

Closing - fixed in RED 0.8.8