Error when response large content using strbuf_set_static #77
Comments
|
This is a pretty curious issue. I've never encountered it before. Is the metadata size correct before |
|
Yes, metadata size is correct. Even after doing strbuf_set_static, I save metadata.size into local file, I can read it well, content is enough. I write a test code with static file, open local file and serve by lwan, with content size 3MB (this image: http://www.zastavki.com/pictures/originals/2013/Love__040998_.jpg), it fail. Source code: static size_t
get_file_size(const char * file_name) {
struct stat sb;
if (stat(file_name, & sb) != 0) {
fprintf(stderr, "'stat' failed for '%s': %s.\n",
file_name, strerror(errno));
exit(EXIT_FAILURE);
}
return sb.st_size;
}
// ham nay load file tu dia len memory
size_t ae_load_file_to_memory(const char *filename, char **result) {
size_t size = 0;
FILE *f = fopen(filename, "rb");
if (f == NULL) {
*result = NULL;
return 0; // -1 means file opening fail
}
fseek(f, 0, SEEK_END);
size = (size_t) ftell(f);
fseek(f, 0, SEEK_SET);
*result = (char *) malloc(size + 1);
if (size != fread(*result, sizeof (char), size, f)) {
free(*result);
return 0; // -2 means file reading fail
}
fclose(f);
(*result)[size] = 0;
return size;
}
// handler trong file config
lwan_http_status_t
static_handler(lwan_request_t *request,
lwan_response_t *response,
void *data __attribute__((unused))) {
struct file_info_t *fi;
fi = calloc(1, sizeof (fi));
lwan_url_map_t *url_map;
url_map = lwan_trie_lookup_prefix(l.url_map_trie, request->original_url.value);
char physical_path[200];
printf("prefix: %s\n", url_map->prefix);
strcpy(physical_path, l.config.root);
strcat(physical_path, request->url.value);
fi = (struct file_info_t*) cache_coro_get_and_ref_entry(cache,
request->conn->coro, (const char*) physical_path);
response->mime_type = lwan_determine_mime_type_for_file_name("a.jpg");
// tra ra cho browser
strbuf_set_static(response->buffer, fi->metadata.content, fi->metadata.size);
return HTTP_OK;
}
// read file and initialize cache object
static struct cache_entry_t *
create_fileinfo(const char *file_path,
void *context __attribute__((unused))) {
struct file_info_t *fileinfo_;
fileinfo_ = calloc(1, sizeof (struct file_info_t));
size_t file_size = 0;
char *content = NULL;
ae_load_file_to_memory(file_path, &content);
if (content) {
file_size = get_file_size(file_path);
fileinfo_->key = file_path;
fileinfo_->metadata.content = content;
fileinfo_->metadata.size = file_size;
return (struct cache_entry_t *) fileinfo_;
}
return NULL;
}
// callback to destroy cache
static void
destroy_fileinfo(struct cache_entry_t *entry,
void *context __attribute__((unused))) {
struct file_info_t *fileinfo = (struct file_info_t *) entry;
if (!fileinfo) return;
free(fileinfo->metadata.content);
//free(fileinfo->key);
free(fileinfo);
}
|
|
From a quick glance, the code looks fine. I'll play with it tonight and see if I can reproduce the issue. |
|
Could not reproduce the bug here, using the JPG file you've provided and similar code (available here). From the perspective of Lwan, the code is working as it should; even The transferred content is the same according to md5sum: Could you please try the linked version and see if it works? There were a few changes to the example code you've posted and some of these changes might do the trick. |
|
Which platform did you make that test? We clone newest lwan source code and use your gist, it fail. (we make tests on ubuntu). |
|
I do all my tests using an up-to-date Arch Linux box, but I'm not sure if that'll make a difference. I've updated the gist with print statements; could you please test it again and paste the output of your program running under This is just a hunch, but might give some clue as well. Here, |
|
Output of strace command
You can see writev returns wrong content size.
|
|
That's really odd. Writes performed by As a last question for today: I'm testing on |
|
I am testing on localhost too. I do this test on 2 different machines (both are Ubuntu) and it makes no difference. |
|
I know what's the problem. Will give a try at fixing it tonight. |
|
Thank :) |
Apparently I misunderstood the affirmation that writes performed by writev() are atomic, and never considered that short writes would be an issue. However, as evidenced by Issue #77, this happens even on localhost. Use the technique shown in this[1] StackOverflow answer, which basically finds which of the entries in the `struct iovec` that the write couldn't be fully performed, adjust the buffer address and length, and try again. [1] http://stackoverflow.com/a/5859890
|
Confirm this patch fixed this issue. |
|
Hi Ipereira, I got an issue with new version of lwan_writev() function. When I test Lwan web server by running: http_load -rate 1000 -sencond 10 http.txt
=> I guess that new implementation of lwan_writev() may affect to CPU, could you please help me to confirm? Thank you very much. |
|
What's the content of your |
|
For your information.
After 10 seconds, http_load ends, Lwan still runs well, CPU % is normal. Then, while Lwan is running free, I do nothing and look at CPU%. Suddenly, 4 CPU% increase to 100%. So if I enter an url, Lwan can not return anything. Finally, I stop Lwan and see that source code is waiting in accept4. => I don't know if this issue comes from writev() system call. I meant if writev has to write a buffer more than twice, there may be some issues with CPU??. looplooplooploop17445 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. ----- Thank you very much for your strong support ----- |
|
Couldn't reproduce here. Could you please share the handler code as well? The original code by @whatvn had a few issues, and if your code is based on that, that could be the reason. I've cleaned it up a bit; it should be more stable, although not completely correct (reading the file to memory, for instance, won't work if short reads happens). However, unless you're obtaining these images from somewhere else than a file system, it's actually better to just use the built-in file serving module. |
|
Hi Ipereira, The issue still occurs. |
|
Hi Ipereira, |
|
The issue is probably the |
|
Thanks for explanation. Hope that day will come soon. :) |
|
@ancuop Could you please build Lwan in debug mode and run it under Valgrind? Please post the entire log here (or in a gist/pastebin). |
|
Hi Ipereira, So sorry that i don't know how to use gist/pastebin. I put log in next comment. |
|
|
|
this issue is still remain even when with latest source. |
|
@whatvn I couldn't reproduce here, despite having tried it for a while. Could you please test your hypothesis by using gdb and providing a backtrace? |
|
Strange, I tried to reproduce this issue this morning but cannot, but I confirm it did happen several times in our environment, even with normal local file serving (not lwan_serv_files, because your module using sendfile() instead of writev). |
|
I'm closing the issue then. If it happens again and it turns out to be reproducible, please open another issue. |
Apparently I misunderstood the affirmation that writes performed by writev() are atomic, and never considered that short writes would be an issue. However, as evidenced by Issue lpereira#77, this happens even on localhost. Use the technique shown in this[1] StackOverflow answer, which basically finds which of the entries in the `struct iovec` that the write couldn't be fully performed, adjust the buffer address and length, and try again. [1] http://stackoverflow.com/a/5859890
We are working on a lwan module that read images from an external service (written using Apache thrift). The specify content is images that user upload, then save to thrift backend, response to web's user by lwan.
Every object can be served fine when its size is smaller than 2.4MB. But when it's larger that than, lwan response with content size = 2.4 MB, even when we sent content length > 2.4M, that's cause content-length mis-match, image received is not enough (it was cut) to 2.4MB and client have to wait until timeout.
response->mime_type = lwan_determine_mime_type_for_file_name(file_name);
strbuf_set(response->buffer,(char*) fi->metadata.content, fi->metadata.size);
when metadata.content has size larger than 2.4MB, problem occurs. Is this a known issue?
I tested when lwan serve files module and it works fine.
The text was updated successfully, but these errors were encountered: