Skip to content
This repository

parse POST data #13

Open
xcir opened this Issue June 05, 2012 · 17 comments

4 participants

Shohei Tanaka Eduardo S. Scarpellini Rubén Romero Kacper Why
Shohei Tanaka
xcir commented June 05, 2012

I have heard that you are looking for a POST data parse VMOD.
I was just making it.

https://github.com/xcir/libvmod-parsepost

Your project is important and interesting.
Let me think what I can do for you.

Eduardo S. Scarpellini
Collaborator
Shohei Tanaka
xcir commented June 05, 2012

Hi, scarpellini

Thank you for your interest.

vmod-curl is very good looks syntax.
I try to emulate the example of it.(in weekend ;-)

I want to ask questions.
decoded data is a good state? or encoded(urlencode)

now, data is urlencoded.

Eduardo S. Scarpellini
Collaborator
Rubén Romero
huayra commented June 06, 2012

Tanaka, Eduardo,

There were some discussions (including PHK) about this under VUG5, but I am not sure what the conclusion was. And, I am still too excited to have an opinion yet :-)

I'll catch up with Kacper later this week and get back to you on this.

Great work you both are doing! Congrats!

R.

Kacper Why
Owner

This is a good thing.

Actually we spoke of encode() decode() stuff as well as post handling on the VUG. Conclusion was a go ahead for a POST module and that the varnish devs can help us expose and make things easier for such a module.

Encoding and decoding of post requests should be done here (as it will never be done inside varnish), so that we can match on specific post vars or multiparts, in the style of https://github.com/fastly/libvmod-urlcode

Ideally, Security.vcl needs to match on

  • the whole request body
  • specific body params
  • all body param names

and I see we are getting there :-)

Many backends take multiple levels of encoding, but security.vcl should be able to detect & kill requests that are obfuscated with many levels of encoding.

Shohei Tanaka
xcir commented June 07, 2012

Hi,

I made to hear you opinion. (encode/decode is not implement yet)
https://github.com/xcir/libvmod-parsepost/tree/experimental

Is the code in preparation ,but will complete on weekend.

example

set req.http.post_key_data = postparse.post_header("key");
set req.http.get_key_data = postparse.get_header("key");
set req.http.cookie_key_data = postparse.cookie_header("key");

set req.http.get_key_alldata = postparse.get_body();
set req.http.cookie_key_alldata = postparse.cookie_body();
set req.http.post_key_alldata = postparse.post_body();

all body param names

how do you want to a interface?

example
1)
call
parsepost.get_key();
ret
"key1,key2,key3"

2)
call
parsepost.get_key();
ret
"key1" (1st call)
"key1" (2nd call)
"" (last call)

I'm sorry if I get it wrong.

Rubén Romero
huayra commented June 12, 2012

It seems you got it right now, at least :-)

http://blog.xcir.net/index.php/2012/06/to-parsing-the-postget-request-and-cookie-header-of-varnish-vmod-parsereq/
https://github.com/xcir/libvmod-parsereq

How does this relate to the Sec.VCL + Varnish Firewall Roadmap/To-Do list?

Rubén Romero

I could arrange for a Roadmap/To-Do list update if you seem it relevant. Let me know.

Kacper Why
Owner

now it looks like encode&decode is on top to tie all of this together

Kacper Why
Owner

ok so we already have urlencode / urldecode in the libvmod_urlcode from @drwilco.
I've changed my mind, basically this module will be simpler if we leave decoding to another module.

The encode/decode functions that are vital for a web application firewall are unicode-normalization functions that give the shortest representation of all codepoints. Code that does this should probably end up in a different vmod. After a little research it seems this utf vmod should be based around utf8proc
(http://www.public-software-group.org/utf8proc )

We'll end up with code like this:

req.http.X-SEC-body = urlcode.decode( unicode.decode( parsereq.post_body ));
req.http.X-SEC-url = urlcode.decode( unicode.decode(req.url));

now to the subject at hand, the functions exposed by parsereq. this will probably end up in a parsereq ticket:

  • get_read_keylist() is of limited value because we cannot iterate in VCL. We would like to iterate over these values from VCL, how can we solve this?
    I guess that answers your question: we need all keys in one string or some iteration callback-to-vcl
    We also need all values in one string. Just having get_body() and post_body() is quite a step ahead!

  • get_header / post_header : They should probably be called get_key and post_key?

and while we are parsing the request and talking about headers:
it would be quite useful to be able to match all HTTP headers, and to iterate over HTTP header values.

cheers & keep up the good work. looking forward to hearing from you!

Shohei Tanaka

Good evening,

I thought how to iterate. (just an idea)

parsereq.addfilter_post(othervmod.filter_a());
parsereq.addfilter_post(othervmod.filter_b());
....
parsereq.execfilter(); //iterate registered filter for all post,get,req.http.*? data.

-vcc and c sample:
https://gist.github.com/3933140

What do you think about this code?

get_header / post_header : They should probably be called get_key and post_key?

Oh... certainly.
I'll think it over. (rename or add)

Rubén Romero
Kacper Why
Owner

@everyone https://github.com/comotion/VSF
@xcir: the best would be some way of writing filters in VCL than you can iterate over all headers or params
@huayra: when it's ready

Rubén Romero

Wow, nice work Kacper.

Really awesome wrapping of all the work that has been done since Security.VCL came about in 2009.
(I fixed the README.rst but you have an issue over at github already)

Shohei Tanaka

--sample
https://gist.github.com/3939606

I Changed to call subroutine.
And, I'm going to do also apply to the iterate for header. (req.http.* and other)
What do you think about this code?

--vmod
https://github.com/xcir/libvmod-parsereq/tree/future-iterate

Kacper Why
Owner

hmm.. will this inline C stuff:
C{
Vmod_Func_parsereq.iterate(sp, "get", (const char*)VGC_function_test2);
}C
become
parsereq.iterate("headers", test2)
or
parsereq.iterate("post", filter_function)
?
cause that could work

Shohei Tanaka

I don't think I can without use of inline-C.
Because, VGC_function_XXX is static function.
And, VMOD data-types does not have a function pointer.
I can't come up with a good idea.

Any ideas?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.