Agent invocation

[vigoo]

Resolves #2459
Resolves #2461 (by completely removing it instead of making it optional)
Resolves #1093

Replaces the old invocation model and APIs with a new, agent oriented one on all levels.

HTTP APIs

Worker-service no longer has the old invoke-and-await and invoke endpoints. The only way to invoke is the new invoke-agent endpoint (already created as part of the bridge SDK epic, now extended/fully implemented)

The launch-new-worker endpoint (creates an agent without invoking anything on it) now accepts invocation context and principal.

Note: further API changes to be expected in a next step where we try to remove the need to explicitly use component-ids when it can be inferred from agent type, but that's intentionally NOT part of this PR.

gRPC APIs

The old invocation gRPC endpoints are removed from both the worker-exeuctor and the worker-service. A single new endpoint is created that matches the invoke-agent http endpoint, directly using agentic types (agent-type, data-value, etc) and supporting trigger, schedule and await together.

All the endpoints that potentially create an agent instance have been extended with an optional principal parameter.

WIT changes

Restructured our WIT packages significantly:

• common types moved to golem:core
• golem:rpc does not exist anymore, became part of golem:agent/host
• All golem packages bumped to be @1.5.0

The old wasm-rpc resource is now agentic - its constructor gets agent-type and data-value, and its methods are invoking agent methods using the agent data model too.

Significant changes to the oplog (described below) are applied in the WIT version of the oplog too, and now we also have two types of oplog entries on WIT level just like in code: oplog-entry and public-oplog-entry. This was an old ticket (#1093) that reduces the data travelling in oplog processor calls. An enrich-oplog-entries host function has been added to convert between them if needed.

Oplog changes

The most important oplog changes:

• ImportedFunctionInvoked has been renamed to HostCall
• ExportedFunctionInvoked and ExportedFunctionCompleted is now AgentInvocationStarted and AgentInvocationFinished
• More importantly all invocation-related oplog entry (the above ones and PendingAgentInvocation) are now using the AgentInvocation and AgentInvocationPayload types to talk about invocations.

AgentInvocation is a sum type describing all the possible invocations that we support now:

• AgentInitialization means the agent's initialize is being called
• AgentMethod calls an arbitrary agent method through the agent invoke interface
• SaveSnapshot and LoadSnapshot are special, non-agentic calls for snapshotting
• ProcessOplogEntries is a special non-agentic call for oplog processor plugins
• ManualUpdate is a placeholder in the invocation queue to execute a snapshot-based update

Executor internals

• The worker executor's internals are now everywhere passing around the above mentioned AgentInvocation and its related types. This becomes lowered to raw wasmtime invocation just at the end of the invocation chain.
• The special wasi-http invocation type has been removed.
• Principal is propagated everywhere it's needed
• RPC, worker-proxy and scheduler all using the new invocation APIs

Others

• Significant improvements to the IntoValue/FromValue derivation macros, and explicit unit test verifying that they produce the same type that's in WIT (for oplog-entry)
• The static moonbit wrapper generator has been completely removed from the CLI (and all templates)
• As the CLI's invoke command no longer works through the static WIT wrapper, we lost the ability to call endpoints with non-component-model parameters (unstructured text/binary and multimodal) as they cannot be encoded using the invoke command's WAVE syntax. Note that we can still invoke these things through the new REPLs (and repl-based "eval" like commands)

Out of scope

The agent-id format does not change by this PR - agent-type name is still kebab-cased and the arguments are using the WAVE syntax extended with custom syntax for unstructured/multimodal. The invocation API itself is also expecting agent type name to be the kebab-cased variant.
The next task in the epic will consider changing these (#2835)

[mschuwalow]

Suggested change

	pub async fn new(_token: &str, _role_arn: &str) -> Self {
	pub async fn new() -> Self {

[mschuwalow]

why switch to a specific image? This is another things that will need to be updated

[vigoo]

The old one (which was Minio::default()) was failing with the updated aws crates (which I updated to reduce some redundant transitive dependencies to try to make the build faster).
Maybe it could have been updated by updating testcontainers_modules instead but that affects more things

[mschuwalow]

Looking at the manifests of the test-components these are still needed, no?

[vigoo]

WIT dirs are not needed anymore. That should be updated in the manifests