Skip to content

Commit

Permalink
improve docs with suggestions
Browse files Browse the repository at this point in the history
  • Loading branch information
marc authored and marc committed Jul 12, 2019
1 parent 28197ad commit ec52575
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 11 deletions.
15 changes: 7 additions & 8 deletions docs/topics/developer-tools.rst
Expand Up @@ -252,15 +252,16 @@ If the handy ``has_next`` element is ``true`` (try loading
`quotes.toscrape.com/api/quotes?page=10`_ in your browser or a
page-number greater than 10), we increment the ``page`` attribute
and ``yield`` a new request, inserting the incremented page-number
into our ``url``.
into our ``url``.

.. _requests-from-curl:

In more complex websites, it could be difficult to easily reproduce the
requests, as we could need to add ``headers`` or ``cookies`` to make it work. In those
cases you can export the requests in `cURL <https://curl.haxx.se/>`_ format, by
right-clicking on it in the network tool and using the
requests, as we could need to add ``headers`` or ``cookies`` to make it work.
In those cases you can export the requests in `cURL <https://curl.haxx.se/>`_
format, by right-clicking on each of them in the network tool and using the
:meth:`~scrapy.http.Request.from_curl()` method to generate an equivalent
request. To do so, just import the :class:`~scrapy.http.Request` class and use this
method to get the desired request::
request::

from scrapy import Request

Expand All @@ -273,12 +274,10 @@ method to get the desired request::
"I0LWJlZGYtMmM0YjgzZmI0MGY0' -H 'Connection: keep-alive' -H 'Referer: http"
"://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'")


Alternatively, if you want to know the arguments needed to recreate that
request you can use the :func:`scrapy.utils.curl.curl_to_request_kwargs`
function and you will get a dictionary with the equivalent arguments.


As you can see, with a few inspections in the `Network`-tool we
were able to easily replicate the dynamic requests of the scrolling
functionality of the page. Crawling dynamic pages can be quite
Expand Down
7 changes: 4 additions & 3 deletions docs/topics/dynamic-content.rst
Expand Up @@ -85,11 +85,12 @@ It might be enough to yield a :class:`~scrapy.http.Request` with the same HTTP
method and URL. However, you may also need to reproduce the body, headers and
form parameters (see :class:`~scrapy.http.FormRequest`) of that request.

As all the major browsers allow to export the requests in `cURL
<https://curl.haxx.se/>`_ format, scrapy incorporates the method
As all major browsers allow to export the requests in `cURL
<https://curl.haxx.se/>`_ format, Scrapy incorporates the method
:meth:`~scrapy.http.Request.from_curl()` to generate an equivalent
:class:`~scrapy.http.Request` from a cURL command. To get more information
visit :ref:`the network tool section <topics-network-tool>`.
visit :ref:`request from curl <requests-from-curl>` inside the network
tool section.

Once you get the expected response, you can :ref:`extract the desired data from
it <topics-handling-response-formats>`.
Expand Down

0 comments on commit ec52575

Please sign in to comment.