Hot protocols improvements (for 662) #724

hhexo · 2018-01-30T16:17:35Z

Closes #662.

This change will require FDKs which handle the json function format to be updated.

This change moves some Fn-specific fields around in the JSON protocol structure, thus improving the handling of Fn-specific fields that would otherwise end up in FN-prefixed headers. The rationale for this is in #662.

Fn-specific stuff (Call ID, request url, etc) that used to be stored in the headers of the models.Call is now instead added by the protocol dispatchers at the right time, so that the model reflects what the user has originally provided. The HTTP protocol dispatcher still creates Fn-prefixed headers, because that's the only way it can pass information to the function, but the JSON protocol can provide the information directly in JSON fields.

hhexo · 2018-01-30T16:54:05Z

I have just noticed that the default format is now missing call ID etc, because it's not going through the dispatchers and it was relying on the Fn-specific headers being set in the model. I'll fix that.

hhexo · 2018-01-30T16:56:17Z

As for the others, there's no change in HTTP and the advertised changes in JSON:

HTTP
====

BEFORE:

Request foo: 1
Req: POST hello
Headers:
key:'Content-Type'  val:[application/x-www-form-urlencoded]
key:'Fn_call_id'  val:[01C53ZD26Q47WG600000000000]
key:'Content-Length'  val:[4]
key:'Accept'  val:[*/*]
key:'Fn_request_url'  val:[http://localhost:8080/r/myapp/fndump]
key:'User-Agent'  val:[curl/7.54.0]
key:'Fn_deadline'  val:[2018-01-30T16:47:04.711Z]
key:'Fn_method'  val:[POST]
Env:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=ac7d5a564da1
FN_PATH=/fndump
FN_FORMAT=http
FN_APP_NAME=myapp
FN_MEMORY=128
FN_TYPE=sync
HOME=/root
Body:
blah

AFTER:

Request foo: 1
Req: POST hello
Headers:
key:'Fn_call_id'  val:[01C53ZNTC0E7WG600000000000]
key:'Fn_deadline'  val:[2018-01-30T16:51:51.600Z]
key:'Fn_method'  val:[POST]
key:'Content-Type'  val:[application/x-www-form-urlencoded]
key:'User-Agent'  val:[curl/7.54.0]
key:'Content-Length'  val:[4]
key:'Accept'  val:[*/*]
key:'Fn_request_url'  val:[http://localhost:8080/r/myapp/fndump]
Env:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=ddomizio-Mac
FN_APP_NAME=myapp
FN_PATH=/fndump
FN_MEMORY=128
FN_FORMAT=http
FN_TYPE=sync
HOME=/root
Body:
blah


JSON
====

BEFORE:

{
"call_id":"01C53ZF8XJ47WGA00000000000"
,"content_type":"application/x-www-form-urlencoded"
,"body":"blah"
,"protocol":{"type":"sync"
,"request_url":"http://localhost:8080/r/myapp/fndump"
,"headers":{"Accept":["*/*"],"Content-Length":["4"],"Content-Type":["application/x-www-form-urlencoded"],"Fn_call_id":["01C53ZF8XJ47WGA00000000000"],"Fn_deadline":["2018-01-30T16:48:17.122Z"],"Fn_method":["POST"],"Fn_request_url":["http://localhost:8080/r/myapp/fndump"],"User-Agent":["curl/7.54.0"]}
}
}

AFTER:

{
"call_id":"01C53ZQ9DTE7WGA00000000000"
,"content_type":"application/x-www-form-urlencoded"
,"type":"sync"
,"deadline":"2018-01-30T16:52:39.786Z"
,"body":"blah"
,"protocol":{"type":"http"
,"method":"POST"
,"request_url":"http://localhost:8080/r/myapp/fndump"
,"headers":{"Accept":["*/*"],"Content-Length":["4"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["curl/7.54.0"]}
}
}

denismakogon · 2018-01-30T17:02:03Z

api/agent/protocol/json.go

+	Type        string           `json:"type"`
+	Deadline    string           `json:"deadline"`
+	Body        string           `json:"body"`
+	Protocol    *CallRequestHTTP `json:"protocol"`


First of all, this has to be documented. As follow-up, it is necessary to open issues against FDKs to adopt new changes.

Added documentation in function-format.md. I will file issues on fdks.

denismakogon · 2018-01-30T17:03:37Z

api/agent/protocol/json_test.go

+	if incomingReq.Deadline != ci.Deadline().String() {
+		t.Errorf("Request Deadline assertion mismatch: expected: %s, got %s",
+			ci.Deadline(), incomingReq.Deadline)
+	}


can you add method verification as well?

Added method and request URL testing too.

hhexo · 2018-01-30T17:32:15Z

I fixed the default protocol too, but there is a change there: the deadline used to be in FN_HEADER_Fn_deadline, whereas now it is in the FN_DEADLINE env var:

DEFAULT
=======

BEFORE:

Env: foo
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=ac7d5a564da1
FN_TYPE=sync
FN_HEADER_Fn_deadline=2018-01-30T16:45:14.619Z
FN_HEADER_Accept=*/*
FN_PATH=/fndump
FN_APP_NAME=myapp
FN_MEMORY=128
FN_CALL_ID=01C53Z9PPB47WG200000000000
FN_METHOD=POST
FN_HEADER_User-Agent=curl/7.54.0
FN_HEADER_Content-Length=4
FN_HEADER_Content-Type=application/x-www-form-urlencoded
FN_FORMAT=default
FN_REQUEST_URL=http://localhost:8080/r/myapp/fndump
HOME=/root
Body
blah

AFTER:

Env: foo
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=ddomizio-Mac
FN_PATH=/fndump
FN_DEADLINE=2018-01-30T17:28:21.772Z
FN_HEADER_Content-Length=4
FN_FORMAT=default
FN_APP_NAME=myapp
FN_MEMORY=128
FN_TYPE=sync
FN_METHOD=POST
FN_REQUEST_URL=http://localhost:8080/r/myapp/fndump
FN_CALL_ID=01C541RN6WE7WG200000000000
FN_HEADER_User-Agent=curl/7.54.0
FN_HEADER_Accept=*/*
FN_HEADER_Content-Type=application/x-www-form-urlencoded
HOME=/root
Body
blah

denismakogon · 2018-01-30T19:06:26Z

Overall LGTM. One major concern, JSON format before this change was meant to be close nearly-compatible to CNCF OpenEvent structure, but with this change, we are about to make a step aside. @treeder made JSON proto as it is right now, so i'd like to know if we really want to make that type of changes.

If so, i recommend to do the following: make JSON protocol interface embeddable, so if we'd like to bring OpenEvent to Fn there would just one thing to adjust - actual data structure to serialize.

treeder · 2018-01-30T19:39:10Z

Our json format won't match openevent, which is fine. Once the dust settles on that, we'll just have another format, like openevent (or I think it's changing to cloudevent now).

treeder · 2018-01-30T19:39:58Z

LGTM

rdallman · 2018-01-31T04:08:10Z

api/agent/protocol/factory.go

+		// something meaningful.
+		// This assumes StartedAt was set to something other than the default.
+		// If that isn't set either, then how many things have gone wrong?
+		// TODO: assert or panic in that case


+1 let's just panic. if our tests don't catch this, our tests need more tests

rdallman · 2018-01-31T04:16:42Z

LGTM. the amendments seem like only additions really at the top level with removal from the header bucket, so it won't hard break really.

hhexo added 2 commits January 30, 2018 14:51

Improve deadline handling in streaming protocols

458824c

Move special headers handling down to the protocols

feb5148

hhexo requested review from rdallman and zootalures January 30, 2018 16:24

denismakogon reviewed Jan 30, 2018

View reviewed changes

hhexo added 3 commits January 30, 2018 17:12

Adding function format documentation for JSON changes

9709f91

Add tests for request url and method in JSON protocol

208b382

Fix protocol missing fn-specific info

49b3f31

This was referenced Jan 30, 2018

Support changes to Fn JSON format fnproject/fdk-node#2

Closed

Support changes to Fn JSON format fnproject/fdk-go#11

Closed

Support changes to the Fn JSON format fnproject/fdk-python#18

Closed

Support changes to the Fn JSON format fnproject/fdk-ruby#4

Closed

Fix import

36dea73

rdallman reviewed Jan 31, 2018

View reviewed changes

rdallman approved these changes Jan 31, 2018

View reviewed changes

Add panic for something that should never happen

7841d96

hhexo merged commit e753732 into master Jan 31, 2018

kojustin deleted the i662-hot-protocol-improvements branch October 19, 2018 23:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hot protocols improvements (for 662) #724

Hot protocols improvements (for 662) #724

hhexo commented Jan 30, 2018

hhexo commented Jan 30, 2018

hhexo commented Jan 30, 2018

denismakogon Jan 30, 2018

hhexo Jan 30, 2018

denismakogon Jan 30, 2018

hhexo Jan 30, 2018

hhexo commented Jan 30, 2018 •

edited

Loading

denismakogon commented Jan 30, 2018

treeder commented Jan 30, 2018

treeder commented Jan 30, 2018

rdallman Jan 31, 2018

rdallman commented Jan 31, 2018

Hot protocols improvements (for 662) #724

Hot protocols improvements (for 662) #724

Conversation

hhexo commented Jan 30, 2018

hhexo commented Jan 30, 2018

hhexo commented Jan 30, 2018

denismakogon Jan 30, 2018

Choose a reason for hiding this comment

hhexo Jan 30, 2018

Choose a reason for hiding this comment

denismakogon Jan 30, 2018

Choose a reason for hiding this comment

hhexo Jan 30, 2018

Choose a reason for hiding this comment

hhexo commented Jan 30, 2018 • edited Loading

denismakogon commented Jan 30, 2018

treeder commented Jan 30, 2018

treeder commented Jan 30, 2018

rdallman Jan 31, 2018

Choose a reason for hiding this comment

rdallman commented Jan 31, 2018

hhexo commented Jan 30, 2018 •

edited

Loading