Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Worker Function Invocation #532

Merged
merged 140 commits into from
May 23, 2024
Merged
Show file tree
Hide file tree
Changes from 109 commits
Commits
Show all changes
140 commits
Select commit Hold shift + click to select a range
c6a35f1
Increase client size
afsalthaj Apr 13, 2024
8f03480
Specific module builds
afsalthaj Apr 13, 2024
ddaec45
Remove golem router
afsalthaj Apr 13, 2024
f89c955
Merge remote-tracking branch 'origin/main'
afsalthaj Apr 14, 2024
a0db910
Update developer docker compose with redis dependency for worker-exec…
afsalthaj Apr 15, 2024
5b68ebd
Merge remote-tracking branch 'origin/main'
afsalthaj Apr 17, 2024
b824c40
Merge remote-tracking branch 'origin/main'
afsalthaj Apr 17, 2024
0401d44
Merge remote-tracking branch 'origin/main'
afsalthaj Apr 18, 2024
934acbb
Merge remote-tracking branch 'origin/main'
afsalthaj Apr 18, 2024
19e58a1
Merge remote-tracking branch 'origin/main'
afsalthaj Apr 19, 2024
fc5078f
Merge remote-tracking branch 'origin/main'
afsalthaj Apr 19, 2024
c9f861b
Merge remote-tracking branch 'origin/main'
afsalthaj Apr 19, 2024
3ddf69e
Experiment wasm-rpc latest
afsalthaj Apr 19, 2024
2c7d028
Merge remote-tracking branch 'origin/main'
afsalthaj Apr 22, 2024
953c244
Merge remote-tracking branch 'origin/main'
afsalthaj Apr 30, 2024
95a83e0
Merge remote-tracking branch 'origin/main'
afsalthaj May 1, 2024
ae55fa6
Merge remote-tracking branch 'origin/main'
afsalthaj May 7, 2024
6568293
Merge remote-tracking branch 'origin/main'
afsalthaj May 8, 2024
7f9c435
Merge remote-tracking branch 'origin/main'
afsalthaj May 9, 2024
edb80f0
Merge remote-tracking branch 'origin/main'
afsalthaj May 13, 2024
62b28c0
Merge remote-tracking branch 'origin/main'
afsalthaj May 15, 2024
3dff24d
Merge remote-tracking branch 'origin/main'
afsalthaj May 15, 2024
df7d4ba
Reduce redundancy in conversion to http response
afsalthaj May 15, 2024
7ff2f55
Revert cargo changes
afsalthaj May 15, 2024
3de89a4
Start building body for each content type
afsalthaj May 15, 2024
8f5b181
Initial draft of content type mapping
afsalthaj May 15, 2024
b66a4b0
Reformat code
afsalthaj May 15, 2024
5a01d2f
Content type
afsalthaj May 16, 2024
318efcc
Handle absence and presence of content type
afsalthaj May 16, 2024
958e963
Reformat code
afsalthaj May 16, 2024
1028b75
Fix code
afsalthaj May 16, 2024
b103b1e
Fix compilation errors
afsalthaj May 16, 2024
5d49e21
Remove unused imports
afsalthaj May 16, 2024
77fa904
Fix contributing.md
afsalthaj May 16, 2024
c99fad6
Get content typoe from response headers
afsalthaj May 16, 2024
bb5caee
Convert to json only if not binary stream for explicit content type a…
afsalthaj May 16, 2024
bccbba4
Make sure response header is set all the time
afsalthaj May 17, 2024
11eea74
Prioritise accept header over response mapping header for content type
afsalthaj May 17, 2024
f890f5a
Fix unused imports
afsalthaj May 17, 2024
6cac723
Reformat code
afsalthaj May 17, 2024
d7fc697
Rename response mapper
afsalthaj May 17, 2024
af85da2
Get refined worker response
afsalthaj May 17, 2024
75aca69
Fix imports
afsalthaj May 17, 2024
ca9ca28
Fix fmt and remove debug lines
afsalthaj May 17, 2024
91f0220
Add comments
afsalthaj May 17, 2024
f43ecee
Support accepting multiple content type
afsalthaj May 17, 2024
0505501
Avoid inspecting request headers
afsalthaj May 17, 2024
4e8f6e6
Fix issues
afsalthaj May 17, 2024
df9c1b9
Fix issues
afsalthaj May 17, 2024
ccf07da
Handle accept headers
afsalthaj May 17, 2024
85c9ffd
Initial draft to handle both accept headers and response headers
afsalthaj May 17, 2024
33002e9
Handle all cases
afsalthaj May 17, 2024
26e6cc2
Reformat code
afsalthaj May 17, 2024
8f81316
Reformat code
afsalthaj May 17, 2024
2f486e1
Reformat code
afsalthaj May 17, 2024
9b2a01a
Add more test cases
afsalthaj May 17, 2024
b6c6831
Complete tests when there is no header content type
afsalthaj May 17, 2024
2f8e4a0
Add more test cases
afsalthaj May 17, 2024
5faac2c
Complete all tests
afsalthaj May 18, 2024
c49ac80
Rename response mapper
afsalthaj May 18, 2024
5651047
Rename to http response body
afsalthaj May 18, 2024
45326b3
Fix comment for accept headers
afsalthaj May 18, 2024
f19e9dc
Merge branch 'main' into support_all_content_type
afsalthaj May 18, 2024
91f5c01
Fix clippy
afsalthaj May 18, 2024
f09017f
Merge remote-tracking branch 'origin/support_all_content_type' into s…
afsalthaj May 18, 2024
ec4c0e2
Remove reundant logic
afsalthaj May 18, 2024
1788c9c
Make accept headers flexible
afsalthaj May 18, 2024
6daced5
Bias towards json
afsalthaj May 18, 2024
298f341
Fix tests
afsalthaj May 18, 2024
af14c12
Try to use same method
afsalthaj May 18, 2024
74ed5b2
Add constraint display
afsalthaj May 18, 2024
7531344
Reduce the number of error types to avoid confusion
afsalthaj May 18, 2024
8dbf5eb
Fix typo
afsalthaj May 19, 2024
35c29c0
Fix docker configurations
afsalthaj May 19, 2024
3c70d3a
Fix user facing docker files
afsalthaj May 19, 2024
d4cd1b4
Merge remote-tracking branch 'origin/main'
afsalthaj May 19, 2024
768ff06
Merge remote-tracking branch 'origin/main'
afsalthaj May 19, 2024
fa31cf1
Merge remote-tracking branch 'origin/main'
afsalthaj May 19, 2024
8f13086
Merge branch 'main' into support_all_content_type
afsalthaj May 20, 2024
a3f7708
Add identifier and string literal into tokenisation
afsalthaj May 20, 2024
1c36c8c
Add identifier
afsalthaj May 20, 2024
87d33af
Start fixing tokenizer test cases
afsalthaj May 20, 2024
ca6a690
Use identifiers all the way
afsalthaj May 20, 2024
3f8deb6
Implement function call
afsalthaj May 21, 2024
a671580
Handle tuple
afsalthaj May 21, 2024
c18e074
Remove request and response details from evaluation context
afsalthaj May 21, 2024
454d54f
Remove concatenated str
afsalthaj May 21, 2024
fd1995b
Create record
afsalthaj May 21, 2024
ace07f9
Create record
afsalthaj May 21, 2024
1717ade
Complete internal function
afsalthaj May 21, 2024
6d4eef2
Start fixing tests
afsalthaj May 21, 2024
37a02ed
Fix last line
afsalthaj May 21, 2024
cc4f225
Fix tokenizer
afsalthaj May 22, 2024
0328490
Fix more tests
afsalthaj May 22, 2024
857fb21
Support double quotes
afsalthaj May 22, 2024
495257e
Fix warnings
afsalthaj May 22, 2024
d031131
Fix more test cases
afsalthaj May 22, 2024
9f57c4d
Fix test cases further
afsalthaj May 22, 2024
cac964f
Fix all tests with 10 remaining
afsalthaj May 22, 2024
c8e9f6a
Fix all tests
afsalthaj May 22, 2024
42e10be
Reformat code
afsalthaj May 22, 2024
b45b048
Merge remote-tracking branch 'origin/main'
afsalthaj May 22, 2024
38dc89c
Merge branch 'main' into bring_context
afsalthaj May 22, 2024
78882d4
Fix clippy
afsalthaj May 22, 2024
167e0e9
Add call function
afsalthaj May 22, 2024
7f235a5
Function calls
afsalthaj May 22, 2024
d439f3e
Implement function call
afsalthaj May 22, 2024
184879d
Async test for resolver
afsalthaj May 22, 2024
5b0e399
Initial implementation on supporting worker function invocation from Rib
afsalthaj May 22, 2024
7b4cc93
Remove the need of functions in binding
afsalthaj May 22, 2024
a0dcd19
Allow worker invocation function
afsalthaj May 23, 2024
ce3c254
Fix intermediate http response
afsalthaj May 23, 2024
fcb12c5
Remove stale comments
afsalthaj May 23, 2024
73493b0
Make mapping for all responses and error to response
afsalthaj May 23, 2024
ba8c4d8
Fix cargo build
afsalthaj May 23, 2024
1a11e16
Fix cargo build
afsalthaj May 23, 2024
d1ff462
Reformat code
afsalthaj May 23, 2024
22da4d9
Start fixing tests
afsalthaj May 23, 2024
f510e5d
Fix tests
afsalthaj May 23, 2024
6cc3a23
Fix tests
afsalthaj May 23, 2024
b280309
Remove GRPC API
afsalthaj May 23, 2024
f234bd0
Handle complex function names with semicolons and slashes
afsalthaj May 23, 2024
fb92e7c
Reformat code
afsalthaj May 23, 2024
962c8e2
Implement test for end to end
afsalthaj May 23, 2024
976cdd3
Add end to end test
afsalthaj May 23, 2024
45e71fc
Make expressions multi line
afsalthaj May 23, 2024
1a77611
Make sure all tests pass
afsalthaj May 23, 2024
8c15ac7
Make sure worker service base is fully built
afsalthaj May 23, 2024
23329eb
Remove unnecessary imports
afsalthaj May 23, 2024
55a048f
Fix clippy
afsalthaj May 23, 2024
f104de6
Make sure worker service fetch metadata of functions
afsalthaj May 23, 2024
8b2238a
Start fixing the APIs
afsalthaj May 23, 2024
c2e1f6c
Fix all APIs
afsalthaj May 23, 2024
fb43406
Fix open API spec
afsalthaj May 23, 2024
0326389
Make response expr mandatory
afsalthaj May 23, 2024
4651499
Fix client and CLI
afsalthaj May 23, 2024
e2243be
Fix client and CLI
afsalthaj May 23, 2024
b26be7d
Fix tests
afsalthaj May 23, 2024
8ccb9db
Try to fix tests
afsalthaj May 23, 2024
3600544
test fixes
justcoon May 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion api-gateway-examples/reusable_open_api_definition.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
"function-name": "golem:it/api/get-cart-contents",
"function-params": [],
"component-id": "2696abdc-df3a-4771-8215-d6af7aa4c408",
"response" : "${{headers : {ContentType: 'json', userid: 'foo'}, {body: worker.response, status: 200}}"
"response" : "${{headers : {ContentType: 'json', userid: "foo"}, {body: worker.response, status: 200}}"
},
"get": {
"summary": "Get Cart Contents",
Expand Down
4 changes: 2 additions & 2 deletions golem-cli/tests/api_definition.rs
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ fn golem_def_with_response(
}

pub fn golem_def(id: &str, component_id: &str) -> HttpApiDefinition {
golem_def_with_response(id, component_id, Some("${{headers: {ContentType: 'json', userid: 'foo'}, body: worker.response, status: 200}}".to_string()))
golem_def_with_response(id, component_id, Some("${{headers: {ContentType: 'json', userid: "foo"}, body: worker.response, status: 200}}".to_string()))
}

pub fn make_golem_file(def: &HttpApiDefinition) -> Result<PathBuf, Failed> {
Expand All @@ -150,7 +150,7 @@ pub fn make_open_api_file(id: &str, component_id: &str) -> Result<PathBuf, Faile
"function-name": "golem:it/api/get-cart-contents",
"function-params": [],
"component-id": component_id,
"response" : "${{headers : {ContentType: 'json', userid: 'foo'}, body: worker.response, status: 200}}"
"response" : "${{headers : {ContentType: 'json', userid: "foo"}, body: worker.response, status: 200}}"
},
"get": {
"summary": "Get Cart Contents",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ impl CustomHttpRequestApi {
}
};

match api_request.resolve(&api_definition) {
match api_request.resolve(&api_definition).await {
Ok(resolved_worker_request) => {
resolved_worker_request
.execute_with::<poem::Response>(&self.worker_request_executor_service)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,7 @@ mod tests {
"function-params": ["${request}"],
"component-id": "00000000-0000-0000-0000-000000000000",
"idempotency-key": "test-key",
"response": "${{headers : {ContentType: 'json', user-id: 'foo'}, body: worker.response, status: 200}}"
"response": "${{headers : {ContentType: \"json\", user-id: \"foo\"}, body: worker.response, status: 200}}"
}))]
.into_iter()
.collect(),
Expand All @@ -312,14 +312,14 @@ mod tests {
Expr::Literal("worker-".to_string()),
Expr::SelectField(
Box::new(Expr::SelectField(
Box::new(Expr::Request()),
Box::new(Expr::Identifier("request".to_string())),
"body".to_string()
)),
"user".to_string()
)
]),
function_name: "test".to_string(),
function_params: vec![Expr::Request()],
function_params: vec![Expr::Identifier("request".to_string())],
component_id: ComponentId(Uuid::nil()),
idempotency_key: Some(Expr::Literal("test-key".to_string())),
response: Some(ResponseMapping(Expr::Record(
Expand All @@ -340,7 +340,7 @@ mod tests {
(
"body".to_string(),
Box::new(Expr::SelectField(
Box::new(Expr::Worker()),
Box::new(Expr::Identifier("worker".to_string())),
"response".to_string(),
)),
),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ mod tests {
vec![
(
"body".to_string(),
Box::new(Expr::Variable("function.return".to_string())),
Box::new(Expr::Identifier("function.return".to_string())),
),
(
"status".to_string(),
Expand All @@ -124,7 +124,7 @@ mod tests {

assert_eq!(
http_response_mapping.body,
Expr::Variable("function.return".to_string())
Expr::Identifier("function.return".to_string())
);
assert_eq!(
http_response_mapping.status,
Expand Down Expand Up @@ -171,7 +171,7 @@ mod tests {
vec![
(
"body".to_string(),
Box::new(Expr::Variable("function.return".to_string())),
Box::new(Expr::Identifier("function.return".to_string())),
),
(
"headers".to_string(),
Expand Down Expand Up @@ -199,7 +199,7 @@ mod tests {
vec![
(
"body".to_string(),
Box::new(Expr::Variable("worker.response".to_string())),
Box::new(Expr::Identifier("worker.response".to_string())),
),
(
"status".to_string(),
Expand Down
98 changes: 80 additions & 18 deletions golem-worker-service-base/src/evaluator/evaluator_context.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,26 +4,20 @@ use crate::evaluator::Getter;
use crate::merge::Merge;
use crate::worker_binding::RequestDetails;
use crate::worker_bridge_execution::{RefinedWorkerResponse, WorkerRequest};
use golem_wasm_ast::analysis::AnalysedFunction;
use golem_wasm_rpc::TypeAnnotatedValue;

// Evaluator of an expression doesn't necessarily need a context all the time, and can be empty.
// or contain worker details, request details, worker_response or all of them.

#[derive(Clone)]
pub struct EvaluationContext {
pub worker_request: Option<WorkerRequest>,
pub worker_response: Option<RefinedWorkerResponse>,
pub variables: Option<TypeAnnotatedValue>,
pub request_data: Option<RequestDetails>,
pub analysed_functions: Vec<AnalysedFunction>,
}

impl EvaluationContext {
pub fn empty() -> Self {
EvaluationContext {
worker_request: None,
worker_response: None,
variables: None,
request_data: None,
analysed_functions: vec![],
}
}

Expand All @@ -46,11 +40,28 @@ impl EvaluationContext {
}

pub fn from_request_data(request: &RequestDetails) -> Self {
let variables = internal::request_type_annotated_value(request);

EvaluationContext {
worker_request: None,
worker_response: None,
variables: None,
request_data: Some(request.clone()),
variables: Some(variables),
analysed_functions: vec![],
}
}

#[allow(unused)]
pub fn from_refined_worker_response(worker_response: &RefinedWorkerResponse) -> Self {
let type_annoated_value = worker_response.to_type_annotated_value();

if let Some(typed_res) = type_annoated_value {
let response_data = internal::create_record("response", typed_res);
let worker_data = internal::create_record("worker", response_data);

EvaluationContext {
variables: Some(worker_data),
analysed_functions: vec![],
}
} else {
EvaluationContext::empty()
}
}

Expand All @@ -59,11 +70,62 @@ impl EvaluationContext {
worker_response: &RefinedWorkerResponse,
request: &RequestDetails,
) -> Self {
EvaluationContext {
worker_request: Some(worker_request.clone()),
worker_response: Some(worker_response.clone()),
variables: None,
request_data: Some(request.clone()),
let mut worker_request_data = internal::worker_request_type_annotated_value(worker_request);

let worker_response_data = internal::worker_response_type_annotated_value(worker_response);

if let Some(worker_response) = worker_response_data {
let worker_data = worker_request_data.merge(&worker_response);
let request_data = internal::request_type_annotated_value(request);

let variables = worker_data.merge(&request_data).clone();

EvaluationContext {
variables: Some(variables),
analysed_functions: vec![],
}
} else {
EvaluationContext::from_request_data(request)
}
}
}

mod internal {
use crate::worker_bridge_execution::{RefinedWorkerResponse, WorkerRequest};
use golem_wasm_ast::analysis::AnalysedType;
use golem_wasm_rpc::TypeAnnotatedValue;

use crate::worker_binding::RequestDetails;

pub(crate) fn request_type_annotated_value(
request_details: &RequestDetails,
) -> TypeAnnotatedValue {
let type_annoated_value = request_details.to_type_annotated_value();
create_record("request", type_annoated_value)
}

pub(crate) fn worker_request_type_annotated_value(
worker_request: &WorkerRequest,
) -> TypeAnnotatedValue {
let typed_value = worker_request.clone().to_type_annotated_value();
create_record("worker", typed_value)
}

pub(crate) fn worker_response_type_annotated_value(
worker_response: &RefinedWorkerResponse,
) -> Option<TypeAnnotatedValue> {
let typed_value = worker_response.to_type_annotated_value();

typed_value.map(|typed_value| {
let response = create_record("response", typed_value);
create_record("worker", response)
})
}

pub(crate) fn create_record(name: &str, value: TypeAnnotatedValue) -> TypeAnnotatedValue {
TypeAnnotatedValue::Record {
typ: vec![(name.to_string(), AnalysedType::from(&value))],
value: vec![(name.to_string(), value)].into_iter().collect(),
}
}
}
Loading
Loading