Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 508 lines (358 sloc) 21.751 kb
c5b4480 @bagder corrected the URL
authored
1 Online: http://curl.haxx.se/docs/httpscripting.html
b094075 @bagder TheArtOfHttpScripting: extended
authored
2 Date: Jan 19, 2011
c1283c5 @bagder tutorial kind of document
authored
3
4 The Art Of Scripting HTTP Requests Using Curl
5 =============================================
6
7 This document will assume that you're familiar with HTML and general
8 networking.
9
10 The possibility to write scripts is essential to make a good computer
11 system. Unix' capability to be extended by shell scripts and various tools to
12 run various automated commands and scripts is one reason why it has succeeded
13 so well.
14
15 The increasing amount of applications moving to the web has made "HTTP
16 Scripting" more frequently requested and wanted. To be able to automatically
17 extract information from the web, to fake users, to post or upload data to
18 web servers are all important tasks today.
19
20 Curl is a command line tool for doing all sorts of URL manipulations and
21 transfers, but this particular document will focus on how to use it when
22 doing HTTP requests for fun and profit. I'll assume that you know how to
23 invoke 'curl --help' or 'curl --manual' to get basic information about it.
24
25 Curl is not written to do everything for you. It makes the requests, it gets
26 the data, it sends data and it retrieves the information. You probably need
27 to glue everything together using some kind of script language or repeated
28 manual invokes.
29
30 1. The HTTP Protocol
31
32 HTTP is the protocol used to fetch data from web servers. It is a very simple
31dc1f4 @bagder spell corrections, changed some wording
authored
33 protocol that is built upon TCP/IP. The protocol also allows information to
c1283c5 @bagder tutorial kind of document
authored
34 get sent to the server from the client using a few different methods, as will
35 be shown here.
36
37 HTTP is plain ASCII text lines being sent by the client to a server to
38 request a particular action, and then the server replies a few text lines
39 before the actual requested content is sent to the client.
40
b094075 @bagder TheArtOfHttpScripting: extended
authored
41 The client, curl, sends a HTTP request. The request contains a method (like
42 GET, POST, HEAD etc), a number of request headers and sometimes a request
43 body. The HTTP server responds with a status line (indicating if things went
44 well), response headers and most often also a response body. The "body" part
45 is the plain data you requested, like the actual HTML or the image etc.
46
47 1.1 See the Protocol
48
49 Using curl's option --verbose (-v as a short option) will display what kind
50 of commands curl sends to the server, as well as a few other informational
51 texts.
52
53 --verbose is the single most useful option when it comes to debug or even
54 understand the curl<->server interaction.
55
56 Sometimes even --verbose is not enough. Then --trace and --trace-ascii offer
57 even more details as they show EVERYTHING curl sends and receives. Use it
58 like this:
59
60 curl --trace-ascii debugdump.txt http://www.example.com/
c1283c5 @bagder tutorial kind of document
authored
61
62 2. URL
63
64 The Uniform Resource Locator format is how you specify the address of a
ae4f824 @bagder added some -c talk, spell checked
authored
65 particular resource on the Internet. You know these, you've seen URLs like
c1283c5 @bagder tutorial kind of document
authored
66 http://curl.haxx.se or https://yourbank.com a million times.
67
68 3. GET a page
69
70 The simplest and most common request/operation made using HTTP is to get a
71 URL. The URL could itself refer to a web page, an image or a file. The client
72 issues a GET request to the server and receives the document it asked for.
ae4f824 @bagder added some -c talk, spell checked
authored
73 If you issue the command line
c1283c5 @bagder tutorial kind of document
authored
74
75 curl http://curl.haxx.se
76
77 you get a web page returned in your terminal window. The entire HTML document
78 that that URL holds.
79
b094075 @bagder TheArtOfHttpScripting: extended
authored
80 All HTTP replies contain a set of response headers that are normally hidden,
81 use curl's --include (-i) option to display them as well as the rest of the
82 document. You can also ask the remote server for ONLY the headers by using
83 the --head (-I) option (which will make curl issue a HEAD request).
c1283c5 @bagder tutorial kind of document
authored
84
85 4. Forms
86
87 Forms are the general way a web site can present a HTML page with fields for
88 the user to enter data in, and then press some kind of 'OK' or 'submit'
89 button to get that data sent to the server. The server then typically uses
90 the posted data to decide how to act. Like using the entered words to search
91 in a database, or to add the info in a bug track system, display the entered
92 address on a map or using the info as a login-prompt verifying that the user
93 is allowed to see what it is about to see.
94
95 Of course there has to be some kind of program in the server end to receive
96 the data you send. You cannot just invent something out of the air.
97
98 4.1 GET
99
100 A GET-form uses the method GET, as specified in HTML like:
101
102 <form method="GET" action="junk.cgi">
103 <input type=text name="birthyear">
104 <input type=submit name=press value="OK">
105 </form>
106
ae4f824 @bagder added some -c talk, spell checked
authored
107 In your favorite browser, this form will appear with a text box to fill in
c1283c5 @bagder tutorial kind of document
authored
108 and a press-button labeled "OK". If you fill in '1905' and press the OK
109 button, your browser will then create a new URL to get for you. The URL will
110 get "junk.cgi?birthyear=1905&press=OK" appended to the path part of the
111 previous URL.
112
113 If the original form was seen on the page "www.hotmail.com/when/birth.html",
114 the second page you'll get will become
115 "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK".
116
117 Most search engines work this way.
118
119 To make curl do the GET form post for you, just enter the expected created
120 URL:
121
6d88d58 @jamesbursa TheArtOfHttpScripting: use long options
jamesbursa authored
122 curl "http://www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK"
c1283c5 @bagder tutorial kind of document
authored
123
124 4.2 POST
125
126 The GET method makes all input field names get displayed in the URL field of
127 your browser. That's generally a good thing when you want to be able to
128 bookmark that page with your given data, but it is an obvious disadvantage
129 if you entered secret information in one of the fields or if there are a
130 large amount of fields creating a very long and unreadable URL.
131
132 The HTTP protocol then offers the POST method. This way the client sends the
133 data separated from the URL and thus you won't see any of it in the URL
134 address field.
135
136 The form would look very similar to the previous one:
137
138 <form method="POST" action="junk.cgi">
139 <input type=text name="birthyear">
7496e87 @bagder updated somewhat
authored
140 <input type=submit name=press value=" OK ">
c1283c5 @bagder tutorial kind of document
authored
141 </form>
142
143 And to use curl to post this form with the same data filled in as before, we
144 could do it like:
145
b094075 @bagder TheArtOfHttpScripting: extended
authored
146 curl --data "birthyear=1905&press=%20OK%20" \
147 http://www.example.com/when.cgi
c1283c5 @bagder tutorial kind of document
authored
148
149 This kind of POST will use the Content-Type
31dc1f4 @bagder spell corrections, changed some wording
authored
150 application/x-www-form-urlencoded and is the most widely used POST kind.
c1283c5 @bagder tutorial kind of document
authored
151
7496e87 @bagder updated somewhat
authored
152 The data you send to the server MUST already be properly encoded, curl will
153 not do that for you. For example, if you want the data to contain a space,
154 you need to replace that space with %20 etc. Failing to comply with this
155 will most likely cause your data to be received wrongly and messed up.
156
47925f3 @bagder Added a new "13. Web Login" chapter
authored
157 Recent curl versions can in fact url-encode POST data for you, like this:
158
6d88d58 @jamesbursa TheArtOfHttpScripting: use long options
jamesbursa authored
159 curl --data-urlencode "name=I am Daniel" http://www.example.com
47925f3 @bagder Added a new "13. Web Login" chapter
authored
160
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
161 4.3 File Upload POST
c1283c5 @bagder tutorial kind of document
authored
162
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
163 Back in late 1995 they defined an additional way to post data over HTTP. It
164 is documented in the RFC 1867, why this method sometimes is referred to as
165 RFC1867-posting.
c1283c5 @bagder tutorial kind of document
authored
166
167 This method is mainly designed to better support file uploads. A form that
168 allows a user to upload a file could be written like this in HTML:
169
170 <form method="POST" enctype='multipart/form-data' action="upload.cgi">
171 <input type=file name=upload>
172 <input type=submit name=press value="OK">
173 </form>
174
175 This clearly shows that the Content-Type about to be sent is
176 multipart/form-data.
177
178 To post to a form like this with curl, you enter a command line like:
179
6d88d58 @jamesbursa TheArtOfHttpScripting: use long options
jamesbursa authored
180 curl --form upload=@localfilename --form press=OK [URL]
c1283c5 @bagder tutorial kind of document
authored
181
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
182 4.4 Hidden Fields
c1283c5 @bagder tutorial kind of document
authored
183
184 A very common way for HTML based application to pass state information
185 between pages is to add hidden fields to the forms. Hidden fields are
186 already filled in, they aren't displayed to the user and they get passed
187 along just as all the other fields.
188
189 A similar example form with one visible field, one hidden field and one
190 submit button could look like:
191
192 <form method="POST" action="foobar.cgi">
193 <input type=text name="birthyear">
31dc1f4 @bagder spell corrections, changed some wording
authored
194 <input type=hidden name="person" value="daniel">
c1283c5 @bagder tutorial kind of document
authored
195 <input type=submit name="press" value="OK">
196 </form>
197
198 To post this with curl, you won't have to think about if the fields are
199 hidden or not. To curl they're all the same:
200
6d88d58 @jamesbursa TheArtOfHttpScripting: use long options
jamesbursa authored
201 curl --data "birthyear=1905&press=OK&person=daniel" [URL]
c1283c5 @bagder tutorial kind of document
authored
202
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
203 4.5 Figure Out What A POST Looks Like
3a588fc @bagder added "4.5 FIGURE OUT WHAT A POST LOOKS LIKE"
authored
204
205 When you're about fill in a form and send to a server by using curl instead
206 of a browser, you're of course very interested in sending a POST exactly the
207 way your browser does.
208
209 An easy way to get to see this, is to save the HTML page with the form on
ae4f824 @bagder added some -c talk, spell checked
authored
210 your local disk, modify the 'method' to a GET, and press the submit button
3a588fc @bagder added "4.5 FIGURE OUT WHAT A POST LOOKS LIKE"
authored
211 (you could also change the action URL if you want to).
212
213 You will then clearly see the data get appended to the URL, separated with a
214 '?'-letter as GET forms are supposed to.
215
c1283c5 @bagder tutorial kind of document
authored
216 5. PUT
217
218 The perhaps best way to upload data to a HTTP server is to use PUT. Then
219 again, this of course requires that someone put a program or script on the
220 server end that knows how to receive a HTTP PUT stream.
221
222 Put a file to a HTTP server with curl:
223
b094075 @bagder TheArtOfHttpScripting: extended
authored
224 curl --upload-file uploadfile http://www.example.com/receive.cgi
c1283c5 @bagder tutorial kind of document
authored
225
47925f3 @bagder Added a new "13. Web Login" chapter
authored
226 6. HTTP Authentication
c1283c5 @bagder tutorial kind of document
authored
227
47925f3 @bagder Added a new "13. Web Login" chapter
authored
228 HTTP Authentication is the ability to tell the server your username and
229 password so that it can verify that you're allowed to do the request you're
230 doing. The Basic authentication used in HTTP (which is the type curl uses by
231 default) is *plain* *text* based, which means it sends username and password
232 only slightly obfuscated, but still fully readable by anyone that sniffs on
233 the network between you and the remote server.
c1283c5 @bagder tutorial kind of document
authored
234
235 To tell curl to use a user and password for authentication:
236
b094075 @bagder TheArtOfHttpScripting: extended
authored
237 curl --user name:password http://www.example.com
7496e87 @bagder updated somewhat
authored
238
239 The site might require a different authentication method (check the headers
240 returned by the server), and then --ntlm, --digest, --negotiate or even
241 --anyauth might be options that suit you.
a07bc79 @yangtse removed trailing whitespace
yangtse authored
242
c1283c5 @bagder tutorial kind of document
authored
243 Sometimes your HTTP access is only available through the use of a HTTP
244 proxy. This seems to be especially common at various companies. A HTTP proxy
245 may require its own user and password to allow the client to get through to
ae4f824 @bagder added some -c talk, spell checked
authored
246 the Internet. To specify those with curl, run something like:
c1283c5 @bagder tutorial kind of document
authored
247
6d88d58 @jamesbursa TheArtOfHttpScripting: use long options
jamesbursa authored
248 curl --proxy-user proxyuser:proxypassword curl.haxx.se
c1283c5 @bagder tutorial kind of document
authored
249
7496e87 @bagder updated somewhat
authored
250 If your proxy requires the authentication to be done using the NTLM method,
1635187 @bagder minor updates
authored
251 use --proxy-ntlm, if it requires Digest use --proxy-digest.
7496e87 @bagder updated somewhat
authored
252
c1283c5 @bagder tutorial kind of document
authored
253 If you use any one these user+password options but leave out the password
254 part, curl will prompt for the password interactively.
255
1635187 @bagder minor updates
authored
256 Do note that when a program is run, its parameters might be possible to see
257 when listing the running processes of the system. Thus, other users may be
258 able to watch your passwords if you pass them as plain command line
259 options. There are ways to circumvent this.
c1283c5 @bagder tutorial kind of document
authored
260
47925f3 @bagder Added a new "13. Web Login" chapter
authored
261 It is worth noting that while this is how HTTP Authentication works, very
262 many web sites will not use this concept when they provide logins etc. See
263 the Web Login chapter further below for more details on that.
264
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
265 7. Referer
c1283c5 @bagder tutorial kind of document
authored
266
1635187 @bagder minor updates
authored
267 A HTTP request may include a 'referer' field (yes it is misspelled), which
268 can be used to tell from which URL the client got to this particular
269 resource. Some programs/scripts check the referer field of requests to verify
270 that this wasn't arriving from an external site or an unknown page. While
271 this is a stupid way to check something so easily forged, many scripts still
272 do it. Using curl, you can put anything you want in the referer-field and
273 thus more easily be able to fool the server into serving your request.
c1283c5 @bagder tutorial kind of document
authored
274
275 Use curl to set the referer field with:
276
b094075 @bagder TheArtOfHttpScripting: extended
authored
277 curl --referer http://www.example.come http://www.example.com
c1283c5 @bagder tutorial kind of document
authored
278
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
279 8. User Agent
c1283c5 @bagder tutorial kind of document
authored
280
281 Very similar to the referer field, all HTTP requests may set the User-Agent
282 field. It names what user agent (client) that is being used. Many
283 applications use this information to decide how to display pages. Silly web
284 programmers try to make different pages for users of different browsers to
285 make them look the best possible for their particular browsers. They usually
286 also do different kinds of javascript, vbscript etc.
287
7d1ef96 @bagder minor corrections
authored
288 At times, you will see that getting a page with curl will not return the same
289 page that you see when getting the page with your browser. Then you know it
290 is time to set the User Agent field to fool the server into thinking you're
291 one of those browsers.
c1283c5 @bagder tutorial kind of document
authored
292
b094075 @bagder TheArtOfHttpScripting: extended
authored
293 To make curl look like Internet Explorer 5 on a Windows 2000 box:
c1283c5 @bagder tutorial kind of document
authored
294
b094075 @bagder TheArtOfHttpScripting: extended
authored
295 curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]
c1283c5 @bagder tutorial kind of document
authored
296
b094075 @bagder TheArtOfHttpScripting: extended
authored
297 Or why not look like you're using Netscape 4.73 on an old Linux box:
c1283c5 @bagder tutorial kind of document
authored
298
b094075 @bagder TheArtOfHttpScripting: extended
authored
299 curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
c1283c5 @bagder tutorial kind of document
authored
300
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
301 9. Redirects
c1283c5 @bagder tutorial kind of document
authored
302
303 When a resource is requested from a server, the reply from the server may
304 include a hint about where the browser should go next to find this page, or a
305 new page keeping newly generated output. The header that tells the browser
306 to redirect is Location:.
307
308 Curl does not follow Location: headers by default, but will simply display
309 such pages in the same manner it display all HTTP replies. It does however
310 feature an option that will make it attempt to follow the Location: pointers.
311
a07bc79 @yangtse removed trailing whitespace
yangtse authored
312 To tell curl to follow a Location:
313
b094075 @bagder TheArtOfHttpScripting: extended
authored
314 curl --location http://www.example.com
c1283c5 @bagder tutorial kind of document
authored
315
316 If you use curl to POST to a site that immediately redirects you to another
6d88d58 @jamesbursa TheArtOfHttpScripting: use long options
jamesbursa authored
317 page, you can safely use --location (-L) and --data/--form together. Curl will
318 only use POST in the first request, and then revert to GET in the following
319 operations.
c1283c5 @bagder tutorial kind of document
authored
320
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
321 10. Cookies
c1283c5 @bagder tutorial kind of document
authored
322
323 The way the web browsers do "client side state control" is by using
324 cookies. Cookies are just names with associated contents. The cookies are
325 sent to the client by the server. The server tells the client for what path
326 and host name it wants the cookie sent back, and it also sends an expiration
327 date and a few more properties.
328
329 When a client communicates with a server with a name and path as previously
330 specified in a received cookie, the client sends back the cookies and their
331 contents to the server, unless of course they are expired.
332
31dc1f4 @bagder spell corrections, changed some wording
authored
333 Many applications and servers use this method to connect a series of requests
ae4f824 @bagder added some -c talk, spell checked
authored
334 into a single logical session. To be able to use curl in such occasions, we
31dc1f4 @bagder spell corrections, changed some wording
authored
335 must be able to record and send back cookies the way the web application
336 expects them. The same way browsers deal with them.
c1283c5 @bagder tutorial kind of document
authored
337
338 The simplest way to send a few cookies to the server when getting a page with
339 curl is to add them on the command line like:
340
b094075 @bagder TheArtOfHttpScripting: extended
authored
341 curl --cookie "name=Daniel" http://www.example.com
c1283c5 @bagder tutorial kind of document
authored
342
343 Cookies are sent as common HTTP headers. This is practical as it allows curl
344 to record cookies simply by recording headers. Record cookies with curl by
6d88d58 @jamesbursa TheArtOfHttpScripting: use long options
jamesbursa authored
345 using the --dump-header (-D) option like:
c1283c5 @bagder tutorial kind of document
authored
346
b094075 @bagder TheArtOfHttpScripting: extended
authored
347 curl --dump-header headers_and_cookies http://www.example.com
c1283c5 @bagder tutorial kind of document
authored
348
6d88d58 @jamesbursa TheArtOfHttpScripting: use long options
jamesbursa authored
349 (Take note that the --cookie-jar option described below is a better way to
350 store cookies.)
7496e87 @bagder updated somewhat
authored
351
c1283c5 @bagder tutorial kind of document
authored
352 Curl has a full blown cookie parsing engine built-in that comes to use if you
353 want to reconnect to a server and use cookies that were stored from a
354 previous connection (or handicrafted manually to fool the server into
355 believing you had a previous connection). To use previously stored cookies,
356 you run curl like:
357
b094075 @bagder TheArtOfHttpScripting: extended
authored
358 curl --cookie stored_cookies_in_file http://www.example.com
c1283c5 @bagder tutorial kind of document
authored
359
6d88d58 @jamesbursa TheArtOfHttpScripting: use long options
jamesbursa authored
360 Curl's "cookie engine" gets enabled when you use the --cookie option. If you
361 only want curl to understand received cookies, use --cookie with a file that
b094075 @bagder TheArtOfHttpScripting: extended
authored
362 doesn't exist. Example, if you want to let curl understand cookies from a
363 page and follow a location (and thus possibly send back cookies it received),
364 you can invoke it like:
3a588fc @bagder added "4.5 FIGURE OUT WHAT A POST LOOKS LIKE"
authored
365
b094075 @bagder TheArtOfHttpScripting: extended
authored
366 curl --cookie nada --location http://www.example.com
3a588fc @bagder added "4.5 FIGURE OUT WHAT A POST LOOKS LIKE"
authored
367
ae4f824 @bagder added some -c talk, spell checked
authored
368 Curl has the ability to read and write cookie files that use the same file
369 format that Netscape and Mozilla do. It is a convenient way to share cookies
b094075 @bagder TheArtOfHttpScripting: extended
authored
370 between browsers and automatic scripts. The --cookie (-b) switch
371 automatically detects if a given file is such a cookie file and parses it,
372 and by using the --cookie-jar (-c) option you'll make curl write a new cookie
373 file at the end of an operation:
ae4f824 @bagder added some -c talk, spell checked
authored
374
b094075 @bagder TheArtOfHttpScripting: extended
authored
375 curl --cookie cookies.txt --cookie-jar newcookies.txt \
376 http://www.example.com
ae4f824 @bagder added some -c talk, spell checked
authored
377
c1283c5 @bagder tutorial kind of document
authored
378 11. HTTPS
379
380 There are a few ways to do secure HTTP transfers. The by far most common
381 protocol for doing this is what is generally known as HTTPS, HTTP over
31dc1f4 @bagder spell corrections, changed some wording
authored
382 SSL. SSL encrypts all the data that is sent and received over the network and
c1283c5 @bagder tutorial kind of document
authored
383 thus makes it harder for attackers to spy on sensitive information.
384
385 SSL (or TLS as the latest version of the standard is called) offers a
386 truckload of advanced features to allow all those encryptions and key
31dc1f4 @bagder spell corrections, changed some wording
authored
387 infrastructure mechanisms encrypted HTTP requires.
c1283c5 @bagder tutorial kind of document
authored
388
31dc1f4 @bagder spell corrections, changed some wording
authored
389 Curl supports encrypted fetches thanks to the freely available OpenSSL
390 libraries. To get a page from a HTTPS server, simply run curl like:
c1283c5 @bagder tutorial kind of document
authored
391
b094075 @bagder TheArtOfHttpScripting: extended
authored
392 curl https://secure.example.com
c1283c5 @bagder tutorial kind of document
authored
393
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
394 11.1 Certificates
c1283c5 @bagder tutorial kind of document
authored
395
396 In the HTTPS world, you use certificates to validate that you are the one
6f53cf9 @bagder s/you you/you/ thanks to hexo6 at wp.pl
authored
397 you claim to be, as an addition to normal passwords. Curl supports client-
398 side certificates. All certificates are locked with a pass phrase, which you
399 need to enter before the certificate can be used by curl. The pass phrase
400 can be specified on the command line or if not, entered interactively when
401 curl queries for it. Use a certificate with curl on a HTTPS server like:
c1283c5 @bagder tutorial kind of document
authored
402
b094075 @bagder TheArtOfHttpScripting: extended
authored
403 curl --cert mycert.pem https://secure.example.com
c1283c5 @bagder tutorial kind of document
authored
404
7496e87 @bagder updated somewhat
authored
405 curl also tries to verify that the server is who it claims to be, by
1635187 @bagder minor updates
authored
406 verifying the server's certificate against a locally stored CA cert
407 bundle. Failing the verification will cause curl to deny the connection. You
6d88d58 @jamesbursa TheArtOfHttpScripting: use long options
jamesbursa authored
408 must then use --insecure (-k) in case you want to tell curl to ignore that
409 the server can't be verified.
1635187 @bagder minor updates
authored
410
411 More about server certificate verification and ca cert bundles can be read
412 in the SSLCERTS document, available online here:
413
414 http://curl.haxx.se/docs/sslcerts.html
7496e87 @bagder updated somewhat
authored
415
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
416 12. Custom Request Elements
417
418 Doing fancy stuff, you may need to add or change elements of a single curl
419 request.
420
421 For example, you can change the POST request to a PROPFIND and send the data
422 as "Content-Type: text/xml" (instead of the default Content-Type) like this:
423
b094075 @bagder TheArtOfHttpScripting: extended
authored
424 curl --data "<xml>" --header "Content-Type: text/xml" \
425 --request PROPFIND url.com
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
426
427 You can delete a default header by providing one without content. Like you
428 can ruin the request by chopping off the Host: header:
429
b094075 @bagder TheArtOfHttpScripting: extended
authored
430 curl --header "Host:" http://www.example.com
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
431
432 You can add headers the same way. Your server may want a "Destination:"
433 header, and you can add it:
434
b094075 @bagder TheArtOfHttpScripting: extended
authored
435 curl --header "Destination: http://nowhere" http://example.com
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
436
47925f3 @bagder Added a new "13. Web Login" chapter
authored
437 13. Web Login
438
439 While not strictly just HTTP related, it still cause a lot of people problems
440 so here's the executive run-down of how the vast majority of all login forms
441 work and how to login to them using curl.
442
443 It can also be noted that to do this properly in an automated fashion, you
444 will most certainly need to script things and do multiple curl invokes etc.
445
446 First, servers mostly use cookies to track the logged-in status of the
447 client, so you will need to capture the cookies you receive in the
448 responses. Then, many sites also set a special cookie on the login page (to
449 make sure you got there through their login page) so you should make a habit
450 of first getting the login-form page to capture the cookies set there.
451
452 Some web-based login systems features various amounts of javascript, and
453 sometimes they use such code to set or modify cookie contents. Possibly they
454 do that to prevent programmed logins, like this manual describes how to...
455 Anyway, if reading the code isn't enough to let you repeat the behavior
456 manually, capturing the HTTP requests done by your browers and analyzing the
457 sent cookies is usually a working method to work out how to shortcut the
458 javascript need.
459
460 In the actual <form> tag for the login, lots of sites fill-in random/session
461 or otherwise secretly generated hidden tags and you may need to first capture
462 the HTML code for the login form and extract all the hidden fields to be able
463 to do a proper login POST. Remember that the contents need to be URL encoded
464 when sent in a normal POST.
465
466 14. Debug
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
467
468 Many times when you run curl on a site, you'll notice that the site doesn't
469 seem to respond the same way to your curl requests as it does to your
470 browser's.
471
472 Then you need to start making your curl requests more similar to your
473 browser's requests:
474
475 * Use the --trace-ascii option to store fully detailed logs of the requests
476 for easier analyzing and better understanding
477
6d88d58 @jamesbursa TheArtOfHttpScripting: use long options
jamesbursa authored
478 * Make sure you check for and use cookies when needed (both reading with
479 --cookie and writing with --cookie-jar)
c1312ca @bagder Added two chapters: Custom Request Elements and Debug.
authored
480
481 * Set user-agent to one like a recent popular browser does
482
483 * Set referer like it is set by the browser
484
485 * If you use POST, make sure you send all the fields and in the same order as
486 the browser does it. (See chapter 4.5 above)
487
488 A very good helper to make sure you do this right, is the LiveHTTPHeader tool
489 that lets you view all headers you send and receive with Mozilla/Firefox
490 (even when using HTTPS).
491
492 A more raw approach is to capture the HTTP traffic on the network with tools
493 such as ethereal or tcpdump and check what headers that were sent and
494 received by the browser. (HTTPS makes this technique inefficient.)
495
47925f3 @bagder Added a new "13. Web Login" chapter
authored
496 15. References
c1283c5 @bagder tutorial kind of document
authored
497
498 RFC 2616 is a must to read if you want in-depth understanding of the HTTP
499 protocol.
500
b094075 @bagder TheArtOfHttpScripting: extended
authored
501 RFC 3986 explains the URL syntax.
c1283c5 @bagder tutorial kind of document
authored
502
503 RFC 2109 defines how cookies are supposed to work.
504
ae4f824 @bagder added some -c talk, spell checked
authored
505 RFC 1867 defines the HTTP post upload format.
506
c1283c5 @bagder tutorial kind of document
authored
507 http://curl.haxx.se is the home of the cURL project
Something went wrong with that request. Please try again.