Microsoft Dynamics 365 CRM integration: Why are leads not created by the application user?

[ericwesterhof]

Question

If you use a Microsoft Dynamics 365 CRM integration to create leads in Dynamics, the CreatedBy property of the lead does not show the application user, but instead the name of the system administrator who gave Formie access to Dynamics (this is the account who made the connection in "Step 3: Test Connection").
The application user is of course linked to the registered app in dynamics, as described in "Step 2. Create an Application User".
My question: Why does the CreatedBy property of the created lead not show the application user, but the name of the above-mentioned system administrator?

Additional context

No response

[engram-design]

It looks like it's possible to set the value, but as the UID for this user isn't something we record with the integration settings, we'll have to get it from somewhere. We'll likely add this as a mappable field for each data object, where you can pick a user from as well.

Added for the next release. To get this early, run composer require verbb/formie:"dev-craft-4 as 2.0.36".

[ericwesterhof]

Thanks! The customer where this occurs still uses Craft 3. Will this solution also be available for the plugin in Craft 3?

[engram-design]

Added for the next release. To get this early, run composer require verbb/formie:"dev-craft-3 as 1.6.32".

[engram-design]

Fixed in 1.6.33

[jamesmacwhite]

Just one thing to bear in mind here just from experience. The user selected will need read permissions on entities like contacts, leads etc in order to be used successfully. You'll know as you'll get something like "Principal user is missing prvReadAccount privilege". This can often be triggered when setting the owner field to a user which doesn't have this.

There is however, a more important potential issue with the createdby field. It is actually not a field valid for create or updating. You can check the entity definition for contact or lead as an example, it's all false (which the mapping does handle by excluding such fields). This means if you try and send a value it will error like the below.

If you need to control the specific user data is created under, you would need to use a specific service account that is then used for authenticating the Formie CRM connection. The user authenticated is important because the API knows this user has access to the required API areas.

You can confirm this by quering the WhoAmI endpoint, the user ID returned is whatever account authenticated the OAuth token. The createdby value is actually quite important and tied to either a CRM Login user = systemuser Entity. It can be licensed user, application user (service account), or SYSTEM (only used by CRM product).

I don't think you can just override this, through mapping. The service account is likely the best approach, but will require configuration at the system administrator level.

{
    "error": {
        "code": "0x80048d19",
        "message": "Error identified in Payload provided by the user for Entity :'contacts', For more information on this error please follow this help link https://go.microsoft.com/fwlink/?linkid=2195293  ---->  InnerException : Microsoft.OData.ODataException: A 'PrimitiveValue' node with non-null value was found when trying to read the value of the property 'createdby'; however, a 'StartArray' node, a 'StartObject' node, or a 'PrimitiveValue' node with null value was expected.\r\n   at Microsoft.OData.JsonLight.ODataJsonLightPropertyAndValueDeserializer.ValidateExpandedNestedResourceInfoPropertyValue(IJsonReader jsonReader, Nullable`1 isCollection, String propertyName, IEdmTypeReference typeReference)\r\n   at Microsoft.OData.JsonLight.ODataJsonLightResourceDeserializer.ReadPropertyWithValue(IODataJsonLightReaderResourceState resourceState, String propertyName, Boolean isDeltaResourceSet)\r\n   at Microsoft.OData.JsonLight.ODataJsonLightResourceDeserializer.<>c__DisplayClass9_0.<ReadResourceContent>b__0(PropertyParsingResult propertyParsingResult, String propertyName)\r\n   at Microsoft.OData.JsonLight.ODataJsonLightDeserializer.ProcessProperty(PropertyAndAnnotationCollector propertyAndAnnotationCollector, Func`2 readPropertyAnnotationValue, Action`2 handleProperty)\r\n   at Microsoft.OData.JsonLight.ODataJsonLightResourceDeserializer.ReadResourceContent(IODataJsonLightReaderResourceState resourceState)\r\n   at Microsoft.OData.JsonLight.ODataJsonLightReader.StartReadingResource()\r\n   at Microsoft.OData.JsonLight.ODataJsonLightReader.ReadResourceSetItemStart(PropertyAndAnnotationCollector propertyAndAnnotationCollector, SelectedPropertiesNode selectedProperties)\r\n   at Microsoft.OData.JsonLight.ODataJsonLightReader.ReadAtStartImplementationSynchronously(PropertyAndAnnotationCollector propertyAndAnnotationCollector)\r\n   at Microsoft.OData.ODataReaderCore.ReadImplementation()\r\n   at Microsoft.OData.ODataReaderCore.InterceptException[T](Func`1 action)\r\n   at System.Web.OData.Formatter.Deserialization.ODataReaderExtensions.ReadResourceOrResourceSet(ODataReader reader)\r\n   at System.Web.OData.Formatter.Deserialization.ODataResourceDeserializer.Read(ODataMessageReader messageReader, Type type, ODataDeserializerContext readContext)\r\n   at System.Web.OData.Formatter.ODataMediaTypeFormatter.ReadFromStream(Type type, Stream readStream, HttpContent content, IFormatterLogger formatterLogger)."
    }
}

I would probably recommend removing the createdby field in the mapping. It will likely cause issues. Certainly from the Web API I don't think you can update this field. I have read some information that suggests the Dynamics 365 SDK can and potentially import processes, but this is a different layer and all Formie related actions are the Web API.

[jamesmacwhite]

There is however a way to impersonate users through the API, you do however still need certain permissions within the environment for it to be possible.

You're still going to need to obtain a valid systemuser ID somewhere and pass in an additional header in the payload when creating records.

https://learn.microsoft.com/en-us/power-apps/developer/data-platform/webapi/impersonate-another-user-web-api

So doing this, the createdby value is set to the impersonated user value provided in the headers and the authenticated user is set as the on behalf of fields.

When an operation such as creating an entity is performed using impersonation, the user who actually performed the operation can be found by querying the record including the createdonbehalfby single-valued navigation property. A corresponding modifiedonbehalfby single-valued navigation property is available for operations that update the entity.

This is top level integration stuff however, because the HTTP headers are defined in the base class, so a impersonation system users list likely needs to exist to be mapped on the core settings, rather than indiviual mapping potentially.

I can confirm with API testing in Postman that it works. I impersonated a random service account which absolutely has nothing to do with the authentication.

I did have to use MSCRMCallerID and not CallerObjectId in the payload, as the latter gave a 403 Forbidden response on the API request.

It is worth pointing out the the application user for authentication is not a valid for this, only CRM licensed users, service accounts and the SYSTEM user.

Despite the impersonation feature being available, you are probably more likely better to just configure a service account and authenticate Formie under that to control the created by value through the API itself. If you were to implement this into Formie, you'd need to have probably 3 additional settings:

1. Impersonate created by user (lightswitch), ability to turn this feature on or off.
2. Header value to set on POST requests when creating records (values: CallerObjectId or MSCRMCallerID)
3. System user ID to pass in as the value.

Getting a list of system users that are actually enabled and licenced for CRM usage has already proved to be a pain with #1579, so this may need to be a string field for a GUID, given you could only set one user value anyway and not all user values may be valid. You'd have to warn that it's an advanced setting and you better know what you are doing if you set it, given it's going to be set across all CRM requests when a POST is concerned and if it's invalid you will cause an error with all API requests.

[engram-design]

So what’s strange is that during my testing this was working to set the created by value and I didn’t get an error back from the API about it.

But maybe we need to add an option at the settings level for use in a header value for the HTTP client.

[jamesmacwhite]

Ah, that's a my bad, helps including @odata.bind! When you've written a helper function to do it for you everywhere, you forget about it when API testing, my apologies. So, not as bad, it won't cause an API error but I don't think it's a valid field that can be set in the payload data, it won't set the value to what it provided I don't think.

Impersonation is the correct way I think, outside of changing the user context the OAuth authentication is run.

[jamesmacwhite]

OK apparently, you can ignore me, as despite being not valid for create or update, you can set it apparently and it does work, just tested it. Makes you wonder what isValidForCreate and isValidforUpdate being false actually does then!

So probably doesn't need to be removed and can be mapped still, however implementing impersonation settings, may be an additional solution to allow it to be controlled across all Formie forms without indiviual mapping.

The main benefit is the setting of the created by on behalf of data keeping a more accurate audit trail I guess.

[engram-design]

Yeah, I can recall seeing that isValidForCreate and isValidforUpdate and it working regardless! I'll look at adding it to the plugin setting.

[jamesmacwhite]

Indeed! Once again, don't trust Microsoft documentation!

Sorry for the noise, I hope at least the impersonation information was useful.

[engram-design]

Appreciate the insight and research. It all shouldn’t be this hard 😭