xmlRegisterFsInputProviders() won't initialize the library and won't work if it is the first call

[veemex]

I have an issue and I don't exactly know from where ir comes, but I can reproduce it.
My stack is node js 20.17.0, express.js and the issues happens on mac and linux as well.
Wherenver the process restarts the first validation fails finding the proper paths to the imports from the schema, the following calls to the validation works. I had to implement a retry logic to bypass the issue for whenever i push an update to production.
It is either me that misuses the library and the imports, or there is indeed something under the hood on how it initializes or waits for stuff.

const path = require("path");
const fs = require("fs").promises;

async function retry(fn, retries = 3, delay = 1000) {
for (let i = 0; i < retries; i++) {
try {
// Try executing the function
return await fn();
} catch (error) {
if (i === retries - 1) {
// If it's the last attempt, rethrow the error
throw error;
}
// Wait for the specified delay before retrying
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
}

async function validateXml(xmlString) {
return await retry(
async () => {
try {
const { XmlDocument, XsdValidator } = await import("libxml2-wasm");
const { xmlRegisterFsInputProviders } = await import("libxml2-wasm/lib/nodejs.mjs");

            xmlRegisterFsInputProviders();

            const schemaDir = path.resolve(__dirname, "..", "configs", "xsd");
            const mainSchemaPath = path.join(schemaDir, "index.xsd");

            // Read the main XSD file as text
            let xsdContent = await fs.readFile(mainSchemaPath, "utf8");

            // Adjust schemaLocation attributes to use absolute paths
            const serializationSchemaPath = path.join(schemaDir, "Serialization.xsd");
            const entitiesSchemaPath = path.join(schemaDir, "Entities.xsd");

            xsdContent = xsdContent.replace(
                /schemaLocation="\.\/Serialization\.xsd"/g,
                `schemaLocation="${serializationSchemaPath}"`
            );
            xsdContent = xsdContent.replace(
                /schemaLocation="\.\/Entities\.xsd"/g,
                `schemaLocation="${entitiesSchemaPath}"`
            );

            // Parse the adjusted XSD content
            const xsdDoc = XmlDocument.fromString(xsdContent, { url: mainSchemaPath });

            // Create a validator from the XSD document
            const validator = XsdValidator.fromDoc(xsdDoc);

            // Handle both single XML string and array of XML strings
            const xmlStrings = Array.isArray(xmlString) ? xmlString : [xmlString];
            const errors = [];

            for (const xmlStr of xmlStrings) {
                const xmlDoc = XmlDocument.fromString(xmlStr);
                try {
                    // Validate the XML document
                    validator.validate(xmlDoc);
                } catch (err) {
                    // Collect validation errors
                    errors.push(err.message);
                } finally {
                    // Dispose of the XML document
                    xmlDoc.dispose();
                }
            }

            // Dispose of the validator and XSD document
            validator.dispose();
            xsdDoc.dispose();

            if (errors.length) {
                throw new Error(`Validation failed: ${errors.join(", ")}`);
            }

            return true;
        } catch (err) {
            console.error("Validation error:", err);
            throw err; // This will trigger the retry
        }
    },
    3,
    150
); // 3 retries, 2000ms delay between attempts

}

module.exports = validateXml;

And the error here:

error:
Validation error:
XmlValidateError: Element '{http://www.w3.org/2001/XMLSchema}element', attribute 'type': References from this schema to components in the namespace 'https://pontaj.s3.eu-central-1.amazonaws.com/Serialization.xsd' are not allowed, since not indicated by an import statement.

at XmlValidateError.fromDetails (file:///.../node_modules/libxml2-wasm/lib/validates.mjs:53:16)
at new XsdValidator (file:///.../node_modules/libxml2-wasm/lib/validates.mjs:158:44)
at XsdValidator.fromDoc (file:///...r/node_modules/libxml2-wasm/lib/validates.mjs:196:28)
at /.../validateXml.js:52:48
at async retry (/.../validateXml.js:8:20)
at async validateXml (/.../validateXml.js:21:12)
at async /.../generareRegistru.js:20:29 {

details: [
{
message: "Element '{http://www.w3.org/2001/XMLSchema}element', attribute 'type': References from this schema to components in the namespace 'https://pontaj.s3.eu-central-1.amazonaws.com/Serialization.xsd' are not allowed, since not indicated by an import statement.\n",
line: 22,
col: 0,
file: '/.../configs/xsd/index.xsd'
}
]
} (validateXml.js:82)

[jameslan]

From the callstack of the error, it failed at the line const validator = XsdValidator.fromDoc(xsdDoc);, which is analyzing the

In the error details, the file name(/.../configs/xsd/index.xsd) looks weird, can you check the value of mainSchemaPath?

In the error message, it is complain the type attribute of an element, and it looks like the content of this type attribute is referencing some type definedin in Serialization. Could you check what this line(line 22 of index.xsd) is?

Also it is possible that the Serialization.xsd is not imported correctly, can you check if the fsInputProvider read it correctly, by finding the node_modules/libxml2-wasm/lib/nodejs.mjs and add console.log() in the match(filename) function (and maybe other functions such as open, read etc) to see whether it reads correctly?

BTW, libxml could handle relative file path in the document/xsd. It will calculate the actually path based on the url you passed in on parsing the parent document.

[veemex]

The path looks weird because i truncated it before posting here, i will follow the steps to try to debug as you mentioned and I will come back to you.

[veemex]

const path = require("path");
const fs = require("fs");

async function retry(fn, retries = 3, delay = 1000) {
for (let i = 0; i < retries; i++) {
try {
// Try executing the function
return await fn();
} catch (error) {
if (i === retries - 1) {
// If it's the last attempt, rethrow the error
throw error;
}
// Wait for the specified delay before retrying
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
}

async function validateXml(xmlString) {
return await retry(
async () => {
try {
const { XmlDocument, XsdValidator, xmlRegisterInputProvider } = await import("libxml2-wasm");
function filePath(filename) {
console.log(filename);
try {
const url = new URL(filename);
return url.protocol === "file:" ? url.pathname : null;
} catch {
return filename;
}
}

            function fileExists(filename) {
                const filepath = filePath(filename);
                return filepath != null && fs.existsSync(filepath);
            }

            const fsInputProviders = {
                match(filename) {
                    console.log("fsInputProviders-match", filename, fileExists(filename));
                    return fileExists(filename);
                },
                open(filename) {
                    const filepath = filePath(filename);

                    console.log("fsInputProviders-open", filepath);

                    if (filepath == null) {
                        return undefined;
                    }
                    try {
                        return fs.openSync(filepath, "r");
                    } catch {
                        return undefined;
                    }
                },
                read(fd, buf) {
                    console.log("fsInputProviders-read");

                    try {
                        return fs.readSync(fd, buf, 0, buf.byteLength, null);
                    } catch {
                        return -1;
                    }
                },
                close(fd) {
                    console.log("fsInputProviders-close");

                    try {
                        fs.closeSync(fd);
                    } catch {
                        // ignore exception in closing file
                    }
                    return true;
                },
            };

            await new Promise((resolve) => setTimeout(resolve, 500)); // 100ms delay

            let result = await xmlRegisterInputProvider(fsInputProviders);
            result = await xmlRegisterInputProvider(fsInputProviders);
            result = await xmlRegisterInputProvider(fsInputProviders);
            result = await xmlRegisterInputProvider(fsInputProviders);
            result = await xmlRegisterInputProvider(fsInputProviders);
            result = await xmlRegisterInputProvider(fsInputProviders);


            const schemaDir = path.resolve(__dirname, "..", "configs", "xsd");
            const mainSchemaPath = path.join(schemaDir, "index.xsd");

            // Read the main XSD file as text
            let xsdContent = await fs.readFileSync(mainSchemaPath, "utf8");

            // Adjust schemaLocation attributes to use absolute paths
            const serializationSchemaPath = path.join(schemaDir, "Serialization.xsd");
            const entitiesSchemaPath = path.join(schemaDir, "Entities.xsd");

            xsdContent = xsdContent.replace(
                /schemaLocation="\.\/Serialization\.xsd"/g,
                `schemaLocation="${serializationSchemaPath}"`
            );
            xsdContent = xsdContent.replace(
                /schemaLocation="\.\/Entities\.xsd"/g,
                `schemaLocation="${entitiesSchemaPath}"`
            );

            // Parse the adjusted XSD content
            const xsdDoc = XmlDocument.fromString(xsdContent, { url: mainSchemaPath });

            // Create a validator from the XSD document
            const validator = XsdValidator.fromDoc(xsdDoc);

            // Handle both single XML string and array of XML strings
            const xmlStrings = Array.isArray(xmlString) ? xmlString : [xmlString];
            const errors = [];

            for (const xmlStr of xmlStrings) {
                const xmlDoc = XmlDocument.fromString(xmlStr);
                try {
                    // Validate the XML document
                    validator.validate(xmlDoc);
                } catch (err) {
                    // Collect validation errors
                    errors.push(err.message);
                } finally {
                    // Dispose of the XML document
                    xmlDoc.dispose();
                }
            }

            // Dispose of the validator and XSD document
            validator.dispose();
            xsdDoc.dispose();

            if (errors.length) {
                throw new Error(`Validation failed: ${errors.join(", ")}`);
            }

            return true;
        } catch (err) {
            console.error("Validation error:", err);
            throw err; // This will trigger the retry
        }
    },
    1,
    150
); // 3 retries, 2000ms delay between attempts

}

module.exports = validateXml;

It might be a problem when dynamic loading the ejs in common.js and wasm maybe ?

From the debugging I've done so far, first time it is imported it does not mantain proper pointers to the functions, it is registering it but they are not properly called:

let registeredCallbacks = null;

export async function xmlRegisterInputProvider(provider) {
// Ensure WASM is ready
if (libxml2.ready) {
await libxml2.ready;
}

// Clean up previous callbacks if they exist
if (registeredCallbacks) {
    try {
        libxml2.removeFunction(registeredCallbacks.matchFunc);
        libxml2.removeFunction(registeredCallbacks.openFunc);
        libxml2.removeFunction(registeredCallbacks.readFunc);
        libxml2.removeFunction(registeredCallbacks.closeFunc);
    } catch (e) {
        console.warn("Error cleaning up previous callbacks:", e);
    }
}

// Create new callbacks
registeredCallbacks = {
    matchFunc: libxml2.addFunction((cfilename) => {
        const filename = libxml2.UTF8ToString(cfilename);
        console.log("matchFunc called with filename:", filename);
        const result = provider.match(filename) ? 1 : 0;
        console.log("matchFunc result:", result);
        return result;
    }, 'ii'),

    openFunc: libxml2.addFunction((cfilename) => {
        const filename = libxml2.UTF8ToString(cfilename);
        console.log("openFunc called with filename:", filename);
        const res = provider.open(filename);
        console.log("openFunc result:", res);
        return res === undefined ? 0 : res;
    }, 'ii'),

    readFunc: libxml2.addFunction((fd, cbuf, len) => {
        console.log("readFunc called with fd:", fd, "length:", len);
        const result = provider.read(fd, libxml2.HEAPU8.subarray(cbuf, cbuf + len));
        console.log("readFunc result:", result);
        return result;
    }, 'iiii'),

    closeFunc: libxml2.addFunction((fd) => {
        console.log("closeFunc called with fd:", fd);
        const result = provider.close(fd) ? 0 : -1;
        console.log("closeFunc result:", result);
        return result;
    }, 'ii')
};

// Register the callbacks
const res = libxml2._xmlRegisterInputCallbacks(
    registeredCallbacks.matchFunc,
    registeredCallbacks.openFunc,
    registeredCallbacks.readFunc,
    registeredCallbacks.closeFunc
);

console.log("xmlRegisterInputCallbacks result:", res);

// If first registration (res === 0) or failed (res < 0), try once more
if (res <= 0) {
    console.log("Initial registration returned 0 or failed, retrying...");
    await new Promise(resolve => setTimeout(resolve, 50));

    const retryRes = libxml2._xmlRegisterInputCallbacks(
        registeredCallbacks.matchFunc,
        registeredCallbacks.openFunc,
        registeredCallbacks.readFunc,
        registeredCallbacks.closeFunc
    );

    console.log("Retry registration result:", retryRes);
    if (retryRes <= 0) {
        throw new Error("Failed to register XML input callbacks after retry");
    }
}

return true;

}

And it is related to the import, because even if I call the function more than once to register, it registers it but still fails as you can see below:

info:
Registering input provider functions... (validateXml.js:82)
info:
xmlRegisterInputCallbacks result:
0 (validateXml.js:82)
info:
Initial registration returned 0 or failed, retrying... (validateXml.js:82)
info:
Retry registration result:
1 (validateXml.js:82)
info:
Registering input provider functions... (validateXml.js:83)
info:
xmlRegisterInputCallbacks result:
2 (validateXml.js:83)
info:
Registering input provider functions... (validateXml.js:84)
info:
xmlRegisterInputCallbacks result:
3 (validateXml.js:84)
info:
Registering input provider functions... (validateXml.js:85)
info:
xmlRegisterInputCallbacks result:
4 (validateXml.js:85)
info:
Registering input provider functions... (validateXml.js:86)
info:
xmlRegisterInputCallbacks result:
5 (validateXml.js:86)
info:
Registering input provider functions... (validateXml.js:87)
info:
xmlRegisterInputCallbacks result:
6 (validateXml.js:87)
info:
finished (validateXml.js:89)
error:
Validation error:
XmlValidateError: failed to load "/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Serialization.xsd": No such file or directory
Element '{http://www.w3.org/2001/XMLSchema}import': Failed to locate a schema at location '/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Serialization.xsd'. Skipping the import.
failed to load "/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Entities.xsd": No such file or directory
Element '{http://www.w3.org/2001/XMLSchema}import': Failed to locate a schema at location '/Use
...
...

And the normal output after I call my route again:

fsInputProviders-match
/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Serialization.xsd
true (validateXml.js:42)
info:
/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Serialization.xsd (validateXml.js:26)
info:
/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Serialization.xsd (validateXml.js:26)
info:
fsInputProviders-open
/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Serialization.xsd (validateXml.js:48)
info:
fsInputProviders-read (validateXml.js:60)
info:
fsInputProviders-read (validateXml.js:60)
info:
fsInputProviders-close (validateXml.js:69)
info:
/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Entities.xsd (validateXml.js:26)
info:
fsInputProviders-match
/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Entities.xsd
true (validateXml.js:42)
info:
/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Entities.xsd (validateXml.js:26)
info:
/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Entities.xsd (validateXml.js:26)
info:
fsInputProviders-open
/Users/mihaitrofin/Documents/GitHub/Ember/ODServer/configs/xsd/Entities.xsd (validateXml.js:48)
info:
fsInputProviders-read (validateXml.js:60)
info:
fsInputProviders-read (validateXml.js:60)
info:
fsInputProviders-close (validateXml.js:69)

[jameslan]

These code won't have different execution (except some misunderstanding) and demonstrates how it runs correctly.

• xmlRegisterInputProvider is not an async function, it returns true or false to indicate if it succeeded.
• When libxml2._xmlRegisterInputCallbacks returns 0, it means it succeeded.
• Executing xmlRegisterInputProvider 6 times registers 6 sets of handlers

Are you able to confirm (via console.log maybe) that before converting xml doc to the validator (where the error happens), the xmlRegisterInputCallbacks was called?

Since it failed on the first call every time? Is it possible to write a test case to demonstrator this issue?

[veemex]

---- deleted link

I attached the files here, you can try and see. I have used the library in a test project, keeping the folder structure and the main validateXml function.
I have run it with Bun as well, to see if there is a difference, same thing happens. You can download, unpack, yarn and node index.js, you will see it happening.

[jameslan]

Found the issue.

It's about initialization correctly.
libxml2._xmlCtxtReadMemory(called by XmlDocument.fromString etc) will call initializer while libxml2._xmlRegisterInputCallbacks (called by xmlRegisterInputProvider and xmlRegisterFsInputProviders) won't.

So if you call XmlDocument.fromString first, the library will be initialized properly and then xmlRegisterFsInputProviders will work.
In the contrast, if callingxmlRegisterFsInputProviders first, then the invocation to XmlDocument.fromString would invalidate the registered handlers. And then the library failed to read the extra schema file via virtual io handlers.

The fix should be initialize correctly before registering the handlers, but there is a workaround for your case:
Calling xmlRegisterFsInputProviders between XmlDocument.fromString and XsdValidator.fromDoc:

• XmlDocument.fromString initialize and parse the xsd file
• xmlRegisterFsInputProviders register input handlers
• XsdValidator.fromDoc creates the validator and read extra xsd files via input handlers.

BTW the path substitution can be skipped because libxml2-wasm could handle the relative path in the xsd.

[jameslan]

Need to find way to initialize the library before any api calls

[jameslan]

Detail of the cause:

• Before calling xmlCtxtReadMemory, xmlNewParserCtxt(parserInternals.c) is called, which calls xmlInitParser(threads.c).
• xmlInitParser calls xmlInitIOCallbacks(xmlIO.c), in which it initializes xmlInputCallbackTable with 1 entry: default matcher.
• If call xmlRegisterInputProvider at the very beginning, before xmlInitIOCallbacks, the callbacks will be the first in the table, and be overwritten by xmlInitIOCallbacks

[veemex]

@jameslan , all right. Thank you for your support! I will implement what you mentioned above in the meantime for a fix. Again, thank you for your time and dedication for maintaining this package!

PS: will edit my comment above and delete the project from there.