Exports with different frequencies

[thomas-tarp]

Hi,

I have a suggestion for a change to the extension.
It changes the export structurally so I would like to discuss and know if there is support for it.

There might also be workarounds or alternative implementations of the intended feature that I have not considered.

Requested Feature
The customer I am working with would like to be able to export various data (tables) with different frequencies.
The rationale is that some data is needed closer to real-time than other data.

Observed Behavior
Currently, the scheduling functionality is bound to an "ADLSE Setup" record where "ADLSE Table" records are lines to to the "ADLSE Setup"-"header", so to speak.

"ADLSE Setup" is implemented following the singleton pattern.
I assume that the rationale behind the singleton implementation is that you want all setup fields on the "ADLSE Setup" record to be the same for all exports, which makes sense.

Suggested Behavior
My immediate idea is to separate the scheduling functionality from the "ADLSE Setup" record and instead introduce a new header entity for exports.

This would make it possible to gather various tables in a collection that can be setup with appropriate configurations, i.e. scheduling in my case.
I also think this would make sense for potential features in the future. One that comes to mind would be the ability to filter the tables that are exported.

Discussion
First of all, it is sensible to consider if it is even relevant to have separate exports with different scheduling configurations.
For example, one might argue that you could just use the highest frequency as the common denominator.
However, that would be based on the assumption that if the data is needed less frequently, it probably also does not change frequently. I believe this will not always be the case.

Secondly, I am not sure if this change would have significant ramifications elsewhere, e.g. for the pipelines or the export itself. I would not think so, but I am not sure.

Let me know what you think.

Best regards!

[DuttaSoumya]

Very good points @thomas-tarp. Would you be interested to cover these points on an online meeting? Kindly write directly to bc2adls@microsoft.com so we can setup one and discuss your idea further.

[Bertverbeek4PS]

@thomas-tarp that is indeed a very nice feature! More customers want this!

[DuttaSoumya]

Thanks for explaining the situation here @thomas-tarp.

Our experience is that pushing the incremental changes into the deltas folder is usually quite quick, even with a daily frequency. We recommend that you consider a high frequency update so that entities that get populated not so often, will not contribute “much” to the export time. In theory, this idea can only be challenged in cases where an entity gets updated in larger volumes and you do not want those changes exported often. The Synapse side of this solution, on the other hand, is more critical in terms of time and cost.

But, it is possible to run exports for tables at different frequencies in the current solution, with slight changes,

• Please consider the API pages created by @Bertverbeek4PS to dynamically change which tables are going to be exported (ADLSETableAPI- allow modifys and enable/ disable them) and then exporting them (ADLSESetupAPI- call StartExport). Thus you may trigger exports from outside BC.
• Alternatively you may directly invoke the codeunit ADLSEExecute with the desired table parameter that you wish to export.
• On the Synapse side, if you wish to consolidate only a few select tables at a different frequency, please consider directly triggering the Consolidation_OneEntity instead of invoking the Consolidation_AllEntities pipeline- you will, of course, have to pass the name of the entity to invoke the consolidation for.

[thomas-tarp]

Thank you as well for the meeting today.

I will test with differentiated frequencies on the Synapse pipelines. It looks like this should fulfill the needs.

I'm closing the issue.