-
Notifications
You must be signed in to change notification settings - Fork 12
Platform Components
Below is a quick synposis for each of the major platform components / capabilities. To see when these capabilities will be available, check the development roadmap.
Read, write and search the TRAC metadata. This allows app developers to start prototyping web-apps that will be built on the TRAC platform. The capabilities for tagging, searching, updates and temporality, including a fully consistent point-in-time view, are largely complete. Definitions of individual object types will evolve as the capabilities using those objects are built in other components.
Includes the TRAC model API for Python and the runtime execution engine. This allows model developers to start building models against the TRAC APIs and execute them in a local sandbox environment. The initial implementation includes the full graph execution engine, long-lived sessions and dev-mode job translation (infers job definitions from local models, for easy sandbox development). Generation of model metadata will also be supported in the initial version.
Read, write and query data from the data layer according to TRAC metadata. This will allow data to be stored and retrieved in apps built against the TRAC web API. The initial implementation will provide read and write capabilities and a local backend for the data plugin API. Query will be added later.
Orchestrate jobs on the platform, using the TRAC metadata to build instructions for the model runtime. Combined with the previous three components, this will allow an end-to-end run, where data and models are loaded into the platform and a job is submitted to use those models and data in the execution of a flow. The results will be available in the data API. The initial implementation will include an implementation of the executor plugin API for local execution. It will also include the API for coordinating jobs between orchestrator instances, also with a local implementation.
Request translation and routing, the current implementation supports gRPC, gRPC-Web, REST/JSON, HTTP proxy and HTTP 1 -> 2 framing. Web sockets support was added in the 0.5 series, for gRPC client-streaming uploads. The gateway provides authentication using a provider mechanism that intercepts incoming requests. Providers can be added using the TRAC plugin mechanism, results from the provider are assembled into JWT which is used in the core platform services.
In live deployments some capabilities of the gateway may be provided by cloud services or standard infrastructure offerings - routing, load balancing etc. It is fine to use some capabilities of the TRAC gateway and not others.
Object-level access control, sign-off, resource allocation and business vocabulary are all managed by the policy service. All these features are configured using policy objects and processed through a generic policy engine, to separate application workflow from the TRAC platform. Separation between applications can be achieved using either logical "applications" in the policy service (which can share resources according to their access control policies) or physically separated tenants (which must import / export resources and use separate service accounts for communicating with back-end resources).
Dynamic SQL push down allows optimized querying over all data held in the TRAC platform, using query capabilities of the underlying technology stack (Big Query, Athena, Data Lake Analytics, HIVE). In-chain overlays allow models or SQL fragments to be inserted into a calculation chain as part of a model run.
Implementation of the platform plugins allows TRAC to use the native services of each platform for data/compute tasks, as well as providing integration into other platform services (e.g. secret management).
Deployable packages for the platform and core plugins are published with every release of TRAC. Packages for model and app developers are available on PyPI and NPM and the platform libraries are available in Maven. Each release also includes a compliance report, with outputs from the compliance scans on all platform components, available on the GitHub releases page.