Be able to "Enhance" entities that were previously ingested with more information

[mknoedel]

THE PROBLEM

There are times where we acquire more information about an entity later on in the integration, that isn't necessarily available at the time that the entity was created. Access information is the most prevent example of this. We ingest a resource, and in a later step, do analysis on that resource to determine what level of public access to that resource is available (see INT-1561#2 as an example).

The only ways to do this using the existing SDK are to either restructure the dependency graph so that the enhancement information is available in the jobState prior to the initial creation of the targetResource (see INT-1142 and INT-1561 as examples), or to put the targeted entity in the jobState as data (jobState.setData(entity), instead of jobState.addEntity(entity)), wait until all the data available for that entities creation is gathered, and then create that entity later (no examples of this method yet).

Both of these two methods are not atomic and do not lend themselves well to our step-metadata documentation structure:

export const cloudAssetSteps: IntegrationStep<IntegrationConfig>[] = [
  {
    id: STEP_IAM_BINDINGS,
    name: 'IAM Bindings',
    entities: [bindingEntities.BINDINGS],
    relationships: [],
    dependsOn: [],
    executionHandler: fetchIamBindings,
  }, ...
]

THE PROPOSAL

Support the ability to enhance entities that have already been ingested earlier in the integration run. This functionality already exists when properties are added to targetEntities with mapped relationships. We should be able to do this same thing without the use of a mapped relationship.

NOTES

Another option could be to add a new property on stepMetadata notifying that an entity needs to be enhanceable, thus making it so that entity will not be uploaded until either all enhancements are gathered or the end of the integration. Example:

export const cloudAssetSteps: IntegrationStep<IntegrationConfig>[] = [
  {
    id: STEP_IAM_CUSTOM_ROLES,
    name: 'IAM Roles',
    entities: [
      {
        resourceName: 'IAM Role',
        _type: IAM_ROLE_ENTITY_TYPE,
        _class: IAM_ROLE_ENTITY_CLASS,
+       enhanceable: true
      },
    ],
    relationships: [],
    executionHandler: createPrincipalRelationships,
  }
]
The reason why we should not pursue this route is because for the main case of this, access control, all resources need to be enhanceable. This would mean that we would need to be storing the entire integration run in memory and not uploading anything until the end of the run. 

[ndowmon]

I definitely think we need this capability - thank you for adding this proposal!

I want to expand on what you've suggested - that this capability already exists on target entities of mapped relationships, and we would like to be able to do this without needing to create a mapped relationship.

Here I think is another case where our proposal from a few weeks back is an appropriate solution (see 4: Internal Target Entity)

[VDubber]

@mknoedel We are reviewing old SDK issues. This issue is a valuable add. Will the new core have the ability to support this?