Proposed RFC Feature: JSON $import Support

[amzn-pratikpa]

Summary:

The goal is to add support for an $import directive in our JSON serializer so that JSON files can import other json files thereby reducing the need for a lot of duplication in our JSON files.

Feature design description:

Import Formats

The following two import formats will be supported:

"A" : {"$import" : "some.json"}

"A" : {"$import": { "file": "some.json", "patch": [...] } }

Handling the $import directive

ImportResolver

The JsonImportResolver classes provides the following static functions:

ResolveImports: This will recursively parse the loaded json document and for each $import directive it encounters, it will call the ResolveImport function of the importer class. This will recursively parse nested imports too.
RestoreImports: This will go through the list of $import directives in the top level json and call the RestoreImport function of the importer class for each one.
This section focuses on importing objects.

BaseImporterObject

We will have a BaseImporterObject class. This class will have the following functions:

ResolveImport: The default implementation of this function will read the JSON file being imported from disk and placed in a JSON object. If there are any patches included with the import, the patch will be placed in a patch object.
ApplyPatch: This will be called after all the nested imports are resolved for the imported object. The patch will be applied to the fully resolved imported object.
CreatePatch: This will be called before RestorImport to create a patch.
RestoreImport: The default implementation will create an $import directive with a JSON merge patch if one was provided.
GetImports: This will be used to retrieve a list of paths(JSON pointers) and associated import name(name of the file/value to be imported) for all the imports. This is used by JsonImportResolver's RestoreImports to retrieve the paths of imports to restore.
GetDependencies: This will list all the imported files/values on which the loaded json file depends. This is required for asset dependency reporting.
Users can inherit the BaseImporterObject class and customize the implementations of ResolveImport, RestoreImport, CreatePatch, and ApplyPatch to suit their specific needs(such as handling custom patch types).

The following sections describe how resolving and restoring will be handled for the $import directive:

ResolveImport

This will be called for each import(including nested imports). The implementation of this function should load the value for the file being imported, copy any patches if available and return the resolved and unpatched value.

ApplyPatch

This will apply the patch to imported value once all the nested imports have been resolved. The default implementation uses json merge patch.

CreatePatch

This will create a patch if required. The default implementation uses json merge patch.

RestoreImport

This will be called for each import in the top level file(i.e., the file being resolved/restored). This will not be called for nested files because if A.json imports B.json which in turn imports C.json, then any changes to the value imported from C should be patched in A.json's import directive. The intermediate import B.json shouldn't need to be restored.

Handling Arrays

Arrays are treated similar to objects for the most part. So if the imported file has an array at its root, it will be imported as a whole into the importing object. Partial arrays are not supported because it's not easy to maintain the order of the imported elements. rapidjson doesn't have an insert function to insert into arrays(not that I know of anyway). So we would have to pop all elements after the import, push all the elements from the import into the array, and then finally push the elements that were there after the import. This is a lot of effort. More simply,
This is supported:

{
    "name" : "valid_array_import",
    "object": {"$import" : "array.json"}
}

This is not supported:

{
    "name" : "invalid_array_import",
    "object": [
        {"$import" : "array.json"},
        {"elem1" : "val1"},
        {"elem2" : "val2"}
    ]
}

In the above example, array.json has an array at its root:

[
    { "element_1" : "value_1" },
    { "element_2" : "value_2" },
    { "element_3" : "value_3" }
]

[santorac]

This should be very useful for Atom's material system, where the PBR material type files currently have roughly 1000 lines of duplicate JSON that are repeated 6 times. There is also a lot of repeated JSON in .shader files.