tool_writeout: add new writeout variable, num_headers #5947

anio · 2020-09-09T15:21:38Z

It's a draft PR.
This variable gives the number of headers. Currently it just work with disabled redirect because I didn't find out how to reset the header counter while redirect.
And also this variable is not present in JSON-formatted writeout because the JSON handler just works with curl_easy_getinfo.

dfandrich · 2020-09-09T15:28:09Z

Where might this be useful? Who would want to know the number of headers without knowing their contents, too?

anio · 2020-09-09T15:57:05Z

I'm using curl to do some sort of mass tests.

This variable may be helpful for example when I'm looking for some caching issues.
Or when I'm looking for routes on a website those are use a different function/server (e.g. proxy_pass)

And also some times it may be helpful to detect soft 404 pages.

p.s. In the next step I will add a new writeout variable to read header value. e.g. %{header[xyz-cache]}.

bagder

There's no docs for the variable
How's it supposed to count amount of headers when one or more redirect are involved?
Test 970 needs update too right?

bagder · 2020-09-09T20:38:06Z

src/tool_writeout.c

+              case VAR_HEADER_COUNT:
+                {
+                  fprintf(stream, "%ld", per->header_count);
+                }


You don't need these braces.

emilengler · 2020-09-10T09:40:31Z

src/tool_cb_hdr.c

+  if(hdrcbdata->config->writeout) {
+    char *value = NULL;
+
+    value = memchr(ptr, ':', cb);


Why do this later. You can also init at variable creation

emilengler · 2020-09-10T09:40:31Z

src/tool_writeout.c

-                if(outs->filename)
-                  fprintf(stream, "%s", outs->filename);
+                if(per->outs.filename)
+                  fprintf(stream, "%s", per->outs.filename);


You can also use fputs. I don't think it's necessary to have the printf overhead here

You are right, but this is a thing of the past and not changes I made, but I will change it.

Sorry but I don't know what you mean

I mean I didn't wrote fprintf(stream, "%s", .

Still something you can change :-)

anio · 2020-09-10T11:22:01Z

1. There's no docs for the variable

I will write soon.

2. How's it supposed to count amount of headers when one or more redirect are involved?

I think same as http_code we should just count the last response header count. I don't know how to do that.

3. Test 970 needs update too right?

Yes, after updating the JSON response to include header_count. I should update tool_writeout_json. As I mentioned before, it currently just works with curl_easy_getinfo.

bagder · 2020-09-10T13:53:11Z

I don't know how to do that.

You get an "empty" header last, don't you? That means the end of the headers for this response.

anio · 2020-09-10T15:14:33Z

You get an "empty" header last, don't you? That means the end of the headers for this response.

Right, but how it can help when I don't know about previous responses? Do I need to count redirects?

bagder · 2020-09-10T15:25:34Z

Shouldn't you just clear the counter when you get a new set of headers after a previous empty header? That way you'd always count only the last set.

ghost · 2020-09-11T13:17:30Z

Congratulations 🎉. DeepCode analyzed your code in 0.318 seconds and we found no issues. Enjoy a moment of no bugs ☀️.

👉 View analysis in DeepCode’s Dashboard | Configure the bot

bagder

Looks fine! How about adding a test with a redirect as well just to make sure it does the right thing then as well?

docs/cmdline-opts/write-out.d

bagder · 2020-09-12T21:38:20Z

docs/cmdline-opts/write-out.d

@@ -39,6 +39,9 @@ option. (Added in 7.26.0)
 The initial path curl ended up in when logging on to the remote FTP
 server. (Added in 7.15.4)
 .TP
+.B header_count
+The count of retrieved headers.


I suggest:: "The number of response headers in the most recent request (restarted at each redirect)"

Should this mention that it doesn't include the first status line?

Should this mention that it doesn't include the first status line?

I think it's not necessary when we are talking about headers.

I've just learned that some users think of that as a (special) header too.

Oops!
By mentioning this, we also have to mention: the status line which includes the status code and reason message IS NOT a header.

So, do you prefer to mention?

I'd prefer to mention it because, as @bagder said, some treat the status line as header line (why???).
If we can be clear on the subject, that we should be.

OK. So, @bagder, what is your opinion?

Being clear is good, which is why I brought this up to begin with.

OK, please give me a good sentence to add.

Maybe like:
"The number of response headers in the most recent request (restarted at each redirect). Note that the status line IS NOT a header."

Thanks! Added.

docs/cmdline-opts/write-out.d

src/tool_writeout_json.c

src/tool_writeout_json.h

This variable gives the number of headers.

bagder · 2020-09-14T22:38:19Z

Thanks!

bagder reviewed Sep 9, 2020

View changes

bagder added cmdline tool HTTP labels Sep 9, 2020

emilengler suggested changes Sep 10, 2020

View changes

anio force-pushed the new-writeout-var-header_count branch from c2d3858 to a88904e Compare September 11, 2020 13:17

anio requested a review from bagder September 12, 2020 09:31

bagder reviewed Sep 12, 2020

View changes

anio force-pushed the new-writeout-var-header_count branch from a88904e to 62616be Compare September 13, 2020 17:08

anio changed the title ~~tool_writeout: add new writeout variable, header_count~~ tool_writeout: add new writeout variable, num_headers Sep 13, 2020

anio requested a review from bagder September 13, 2020 17:09

bagder reviewed Sep 14, 2020

View changes

docs/cmdline-opts/write-out.d Outdated Show resolved Hide resolved

src/tool_writeout_json.c Outdated Show resolved Hide resolved

src/tool_writeout_json.c Outdated Show resolved Hide resolved

src/tool_writeout_json.h Outdated Show resolved Hide resolved

anio force-pushed the new-writeout-var-header_count branch from 62616be to 4711135 Compare September 14, 2020 09:53

anio requested a review from bagder September 14, 2020 10:02

bagder approved these changes Sep 14, 2020

View changes

tool_writeout: add new writeout variable, num_headers

695b59b

This variable gives the number of headers.

anio force-pushed the new-writeout-var-header_count branch from 4711135 to 695b59b Compare September 14, 2020 12:52

bagder closed this in 0c1e767 Sep 14, 2020

tool_writeout: add new writeout variable, num_headers #5947

tool_writeout: add new writeout variable, num_headers #5947

anio commented Sep 9, 2020

dfandrich commented Sep 9, 2020 via email

anio commented Sep 9, 2020 •

edited

bagder left a comment

bagder Sep 9, 2020

emilengler Sep 10, 2020

emilengler Sep 10, 2020

anio Sep 10, 2020

emilengler Sep 10, 2020

anio Sep 11, 2020

emilengler Sep 11, 2020

anio commented Sep 10, 2020

bagder commented Sep 10, 2020

anio commented Sep 10, 2020

bagder commented Sep 10, 2020

ghost commented Sep 11, 2020 •

edited by ghost

bagder left a comment

bagder Sep 12, 2020

anio Sep 13, 2020

bagder Sep 13, 2020

anio Sep 13, 2020

jzakrzewski Sep 14, 2020

anio Sep 14, 2020

bagder Sep 14, 2020

anio Sep 14, 2020

jzakrzewski Sep 14, 2020

anio Sep 14, 2020

bagder commented Sep 14, 2020

tool_writeout: add new writeout variable, num_headers #5947

tool_writeout: add new writeout variable, num_headers #5947

Conversation

anio commented Sep 9, 2020

dfandrich commented Sep 9, 2020 via email

anio commented Sep 9, 2020 • edited

bagder left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anio commented Sep 10, 2020

bagder commented Sep 10, 2020

anio commented Sep 10, 2020

bagder commented Sep 10, 2020

ghost commented Sep 11, 2020 • edited by ghost

👉 View analysis in DeepCode’s Dashboard | Configure the bot

bagder left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bagder commented Sep 14, 2020

anio commented Sep 9, 2020 •

edited

ghost commented Sep 11, 2020 •

edited by ghost