Extract resource loading

[peterquiel]

Description

Introduces ResourceLoader and a ResourceLoaderRegistry that enables resource loading from other locations or pre process any loaded resources.

I added an UrlResourcesLoader that loads resources like feature file via http.

[ptrthomas]

would like @edwardsph to review as well.

one question I have is getResource(wd, path) going to come into the picture internally for every attempt to load a file, which is ok - just that I have some "app server" plans (RequestHandler etc) and this may impact performance slightly, perhaps we use a cache or a slightly different model (like we had the hard-coded class-loader previously)

is unregister ever used ?

[edwardsph]

I've grabbed the PR so I can review but I'm afraid it might not be until Thursday due to other commitments.

[peterquiel]

This implementation won't have any performance impact compared to the previous implementation. I just put that Registry in between.
If someone implements a slow ResourceLoader,.. well, that's another story.

The unregister method isn't used from karate core. It's meant to be used by consumers that implement their own resource loader.

UseCase:
You just want to have a special ResourceLoader for one TestSuite and not for the complete test run.
I added it, because every time I see a register method, I would expect that there is is an unregister as well.

[peterquiel]

@edwardsph Keep in mind that you have to register the resource loader before you are able to load feature files from a remote location:

ResourceLoaderRegistry.register(new UrlResourceLoader())

[edwardsph]

Sorry it took a while for me to look at this. I think this code is good and I've been looking at how to use it to load remote features. As a temporary step I added the UrlResourceLoader to the defaults in ResourceLoaderRegistry. Then I went into ScenarioFileReader and changed the following:

    public Resource toResource(String path) {
        if (isClassPath(path) || isRemotePath(path)) {  // add isRemotePath here
            return ResourceUtils.getResource(featureRuntime.suite.workingDir, path);
        } else if (isFilePath(path)) {
            return ResourceUtils.getResource(featureRuntime.suite.workingDir, removePrefix(path));
        } else if (isThisPath(path)) {
            return featureRuntime.resolveFromThis(removePrefix(path));
        } else {
            return featureRuntime.resolveFromRoot(path);
        }
    }

    private static boolean isRemotePath(String text) {
        return text.startsWith("http:") || text.startsWith("https:");
    }

I was then able to successfully run:

Scenario:
* def remote = call read('http://localhost:8080/remote-resource.feature')
* print remote.helper()

I got the remote output as hoped. Next I need to try and load another remote feature from within my remote one (as a sibling e.g. http://localhost:8080/remote-resource-sibling.feature).

I used a local http server for my testing as I was not sure how to set up a MockServer to host the 2 feature files. I was looking at KarateMockHandlerTest and wondering if I could set up 2 background scenarios to represent the 2 remote features, then run a scenario that calls the first remote which calls the second. Can you point me in the right direction on this please?

[ptrthomas]

@edwardsph going offline now, but if you need 2 mocks use Java and manage the ports: https://github.com/intuit/karate/blob/develop/karate-core/src/test/java/com/intuit/karate/core/mock/MockTest.java

[edwardsph]

I'm not sure how to best contribute to this PR. The only way I could think of was either a replacement PR or ask you to push the original changes so I can add a new PR. For now I thought I would describe additions in this comment but if there is a better way, please let me know.

• I added the URLResourceLoader as an additional default loader.
• In ScenarioFileReader I added the changes mentioned above
• In src/test/java/com/intuit/karate/resource I added the following files:

karate-config.js

function fn() {
  var port = karate.properties['karate.server.port'] || 8080;
  var prefix = karate.properties['karate.ssl'] ? 'https' : 'http';
  if (prefix === 'https') {
    karate.configure('ssl', true);
  }
  return {
    mockServerUrl: prefix + '://localhost:' + port
  }
}

call-remote.feature

@ignore
Feature:

Scenario:
  * def remote = call read(`${mockServerUrl}/remote-helper.feature`)
  * match remote.helper() == 'Helped'

remote-features.feature (print statements left in for debugging only)

Feature:

  Scenario: pathMatches('/remote.feature')
    * text response =
    """
    Feature:
    Scenario:
      * print "REMOTE"
      * def a = { remote: true }
      * assert a.remote
    """

  Scenario: pathMatches('/remote-call-remote.feature')
    * text response =
    """
    Feature:
    Scenario:
      * print "CALL REMOTE"
      * def remote = call read(`${mockServerUrl}/remote-helper.feature`)
      * match remote.helper() == "Helped"     
    """

  Scenario: pathMatches('/remote-call-relative.feature')
    * text response =
    """
    Feature:
    Scenario:
      * print "CALL REMOTE SIBLING"
      * def remote = call read('remote-helper.feature')
      * match remote.helper() == "Helped"
    """

  Scenario: pathMatches('/remote-helper.feature')
    * text response =
    """
    Feature:
    Scenario:
      * print "HELPER"
      * def helper = function() { return 'Helped' }
    """

RemoteScenarioTest.java

package com.intuit.karate.resource;

import com.intuit.karate.Results;
import com.intuit.karate.Runner;
import com.intuit.karate.core.MockServer;
import com.intuit.karate.http.HttpServer;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import static org.junit.jupiter.api.Assertions.assertEquals;

/**
 *
 * @author edwardsph
 */
class RemoteScenarioTest {

    static final Logger logger = LoggerFactory.getLogger(RemoteScenarioTest.class);

    static String mockServerUrl = null;

    static HttpServer startMockServer() {
        MockServer server = MockServer.feature("classpath:com/intuit/karate/resource/remote-features.feature").build();
        System.setProperty("karate.server.port", server.getPort() + "");
        mockServerUrl = "http://localhost:" + server.getPort();
        return server;
    }

    @BeforeAll
    static void beforeAll() {
        startMockServer();
    }

    @Test
    void testLocalCallingRemote() {
        Results results = Runner.path("classpath:com/intuit/karate/resource/call-remote.feature")
                .configDir("classpath:com/intuit/karate/resource")
                .tags("~@ignore").parallel(1);
        assertEquals(0, results.getFailCount(), results.getErrorMessages());
    }

    @Test
    void testRemote() {
        Results results = Runner.path(mockServerUrl + "/remote.feature")
                .configDir("classpath:com/intuit/karate/resource")
                .tags("~@ignore").parallel(1);
        assertEquals(0, results.getFailCount(), results.getErrorMessages());
    }

    @Test
    void testRemoteCallingRemote() {
        Results results = Runner.path(mockServerUrl + "/remote-call-remote.feature")
                .configDir("classpath:com/intuit/karate/resource")
                .tags("~@ignore").parallel(1);
        assertEquals(0, results.getFailCount(), results.getErrorMessages());
    }

    @Test
    @Disabled // this will fail since the feature runtime attempts to resolve it based on the temporary file location
    void testRemoteCallingSibling() {
        Results results = Runner.path(mockServerUrl + "/remote-call-relative.feature")
                .configDir("classpath:com/intuit/karate/resource")
                .tags("~@ignore").parallel(1);
        assertEquals(0, results.getFailCount(), results.getErrorMessages());
    }
}

The first 3 tests pass, demonstrating:

• A scenario can call a remote feature
• A runner can execute a remote feature
• That remote feature can then call another remote feature by a fully qualified URL

This is really promising but I think there is more to do:

• Are there any issues pulling remote non-feature resources?
• Can we make a remote feature with a call to relative resource build a URL based on the original URL of the feature? This is what the 4th test is for.

My ideas here are:

• Create a RemoteResource class extending FileResource
• The UrlResourceLoader will create a RemoteResource and pass the URL into it - it's resolve method will use that URL and the path passed in to resolve a URL for the new resource, fetch it and create the temporary file
• When ScenarioFileReader finds a relative resource if calls featureRuntime.resolveFromRoot(path) - this would call the new RemoteResource.resolve method
• I think it should also be possible to use this: but I'm less sure about classpath:

Do you foresee any issues with this or does it look like it would work?

[edwardsph]

Just thinking some more about this I remembered @peterquiel suggesting:

Since you have the loaded feature file at hand you are able to pre process it.
When you load the feature file from a remote location, you can replace any relative path in that feature with an absolute http path.

That would be a much easier solution and wouldn't need the RemoteResource class - just a pre-processing stage whilst the temp file is created in UrlResourceLoader. I'll try that out.

[ptrthomas]

closing because this is stale / out of date and may need re-work