Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is it possible to avoid reallocation during chunked data processing? #48

Open
xdevelnet opened this issue Feb 20, 2019 · 0 comments
Open

Comments

@xdevelnet
Copy link

For example, if i'm receiving huge 200mb file in "multipart/form-data" and i don't want to expand buffer to such enormous size. What I could do?

I was looking at #14 but it seems to be not an appropriate solution in my case. In comparison to example in README.md you just removed part with realloc() but will function process data properly in that case? I don't get it.

The only one solution that comes to my head it to:

  1. read() and immediately write() all incoming data to temprorary file in disk
  2. mmap() whole file to memory
  3. pass whole mmap()ped region to phr_decode_chunked() and decode everything with one pass
  4. write result(s) to files
  5. remove temp file

That's kinda stable and memory-tolerant solution, but i wish there was a way to avoid redundant disk read()/write() and avoid execution of background tasks.

P.S. I'd like ask one more question. I want to use your library with FastCGI, where http headers are already parsed by web server and available via FastCGI api. Is it possible to use phr_parse_response() without previous using phr_parse_headers()?

Before i found your library i was almost stared own small library for parsing POST and POST in multipart form data, which is quite hard if you are using small and fixed size buffer. But I really don't want to do that :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant