How to Handle Asynchronous Scoping

[ballcoach12]

I am working on a scope provider that will load an imported scope. It will do so by calling a remote endpoint via a network call and asking for the code in the imported package. It will then parse the package contents and provide the scope. The problem I'm seeing is that parsing is an asynchronous activity and the ScopeProvider interface expects a return type of Scope not Promise<Scope>. So I am not sure how to structure things so that I can obtain the AST for the imported package.

Is there a recommended way to do this?

[ballcoach12]

After looking at this a bit further, it appears that all I need to do is obtain the LangiumDocument , which can be parsed as shown in the following class method:

private getDocument(input: string, packageName:string): LangiumDocument<DtsPackage> {
        const uri = URI.parse(packageName);
        const result = this.services.shared.workspace.LangiumDocumentFactory.fromString<DtsPackage>(input, uri);
       return result;
    }

I assume that once I get this document, I can run validation on it to make sure it's valid.

[msujew]

Parsing is synchronous in Langium, but loading something from a remote resource likely isn't. How we usually do this (i.e. additional resources from something like a network resource), is by overriding the WorkspaceManager#loadAdditionalDocuments method and loading the resources as soon as we load our workspace. See also the documentation on built-in libraries.

[msujew]

Oh, yeah I always forget that update assumes those documents need to be reparsed. You should use DocumentBuilder#build instead. Note that the created documents should always be part of the LangiumDocuments service. You can always add them via addDocument.

[ballcoach12]

I was able to pull off what you suggest, @msujew. I don't have to make a call to update() on documentBuilder at all. Here is the final solution for reference:

export function addUnresolvedImportCollectionPhase(services: BlueframeDataTransformationServicesServices): void {
    const documentBuilder = services.shared.workspace.DocumentBuilder;
    documentBuilder.onBuildPhase(DocumentState.IndexedContent, async (docs) => {
        // findUnresolvedImports returns only URIs for packages we don't already have
        // It is a TODO to determine when a deployed package changes and how we invalidate and reload it
        const imports = findUnresolvedImports(docs, services);
        for (const imp0rt of imports) {
            const document = await services.references.ImportScopeResolver.getDocument(imp0rt);
            services.shared.workspace.LangiumDocuments.addDocument(document);
        }        
    })
}

[msujew]

Ha, interesting. I'm a bit confused, since these documents shouldn't get indexed. But if it works, it works ¯\_(ツ)_/¯

[ballcoach12]

I'll double-check that. I've checked my global scope computation, though, and the elements are there for sure, and references to elements in the imported documents get resolved. My ImportScopeResolver.getDocument() method looks like this:

async getDocument(uri: URI): Promise<LangiumDocument<DtsPackageRoot>> {
        // a real provider will make a call to the function registry to obtain the document
        // at the given URI.  Here, we just fake in a package
        const documentBuilder = this.services.shared.workspace.DocumentBuilder;           
        const document = this.services.shared.workspace.LangiumDocumentFactory.fromString<DtsPackageRoot>(sally, uri);
        await documentBuilder.build([document], {validation: true});
        return document;
    }

The provider shown here fakes in some code to import, but this gives you an idea of how I'm handling the docs.

[msujew]

Oh I see. Calling documentBuilder.build will of course build those documents. Yeah, that makes sense then. I thought you didn't call the build method at all. 👍

[ballcoach12]

@msujew, a couple of questions. Sorry for my lack of expertise here.

1. What would be the criteria for the findUnresolvedImports() function? In other words, what constitutes an unresolved import in this case?
2. What is the purpose of the newDocuments variable? I don't see it used here, and it isn't clear to me what its type would be. I assume that it's a LangiumDocument.

Also, my general design on this feature is to follow the methodology described in the docs here. I assume that this is the approach that would be recommended for my case, with the exception of how I retrieve the resource for reference from my remote endpoint.

[msujew]

What would be the criteria for the findUnresolvedImports() function? In other words, what constitutes an unresolved import in this case?

An unresolved import points to a document (or multiple documents) that hasn't been loaded yet.

What is the purpose of the newDocuments variable? I don't see it used here, and it isn't clear to me what its type would be. I assume that it's a LangiumDocument.

I've updated the code snippet above. There was a small error. It was supposed to be used for the documentBuilder.update call.