Support external and domain-specific code integrations

[xuang7]

Currently, some customized external or domain-specific code is maintained outside the core platform. Supporting these integrations in a consistent and maintainable way could reduce duplication, divergence, and additional maintenance overhead.

The challenge is that this code may appear in several forms:

• Locally modified third-party code: external code adapted primarily to access data on the platform; these changes may prevent the use of the originally published container images
• Standalone microservices: external or domain-specific services deployed alongside the main platform

These integrations are often implemented independently. Some resulting changes may be highly domain-specific and may not be necessary or broadly useful within the core platform.

For example, the architecture includes an additional domain-specific microservice:

One question is whether Texera should define a standard, documented API for external code and services to access platform-managed data, rather than relying on custom integration logic for each use case.

Feedback on how to support and manage these different types of external and domain-specific code would be very helpful. Please feel free to share any thoughts, concerns, or suggestions. Thank you!

[chenlica]

@xuang7 Please show a few examples with screenshots. Show the architecture of the existing solution.

[aicam]

I don't think its a good idea to add every 3rd party repo to our repo for the following reasons:
1- Every line of code in our main branch is our responsibility, if we keep adding other team code to our, we are adding responsibility of other teams to ours
2- We can not update the 3rd party code
Therefore, I suggest we come up with a patching solution which means to define a process in which some other 3rd party codes can be patched into the application.

[sshiv012]

I may be misunderstanding the intended context, but is this about supporting domain-specific or “flavored” Texera deployments? I am thinking of models such as PostgreSQL’s extension ecosystem, or AsterixDB-based projects such as Graphix and Big Active Data, where domain-specific functionality is maintained separately from the core system. AsterixDB also documents a mechanism for including external extension modules in custom binary assemblies as bill of materials (BOM)

If this is the intended use case, a patching approach makes sense. Each flavor could live in a separate repository containing:

• A pinned Texera version.
• A versioned patch series.
• Additional services or containers.
• Domain-specific configuration.
• CI verifying that the patches apply and the deployment builds.

This would keep domain-specific maintenance outside the Texera repository while making each deployment reproducible. Is that close to the integration model being proposed?

[chenlica]

@aicam I agree we don't want to include third-party code into the Texera codebase.

@sshiv012 : The PostgreSQL’s extension ecosystem and AsterixDB-based projects are indeed something we can learn from to make each Texera deployment extensible.

@xuang7 : please add 1-2 examples to make the need more clear.