Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simple AJAX request #54

Open
stdex opened this issue Oct 15, 2015 · 1 comment
Open

Simple AJAX request #54

stdex opened this issue Oct 15, 2015 · 1 comment

Comments

@stdex
Copy link

stdex commented Oct 15, 2015

Hello. I'm try to emulate simple ajax request.

Scheme:
image

It' must return json:
{phone: "8 xxx xxx-xx-xx"}

But return None for me.
I'm suggest that the cause - cookies. Can someone help me with it?

# -*- coding: utf-8 -*-
import requests
import os, re, json, csv, sys
from robobrowser import RoboBrowser

class Aggregator(object):

    def __init__(self, config):
        main_url, output_file = [config.get(k) for k in sorted(config.keys())]
        self.main_url = main_url
        self.output_file = output_file

    def start_process(self):

        work_url = "https://m.avito.ru/sankt-peterburg/predlozheniya_uslug/almaznoe_burenie_almaznaya_rezka_usilenie_79225740"

        session = requests.Session()
        session.headers.update({
            ':host': 'm.avito.ru',
            ':method': 'GET',
            ':path': '/sankt-peterburg/predlozheniya_uslug/almaznoe_burenie_almaznaya_rezka_usilenie_79225740',
            ':scheme': 'https',
            ':version': 'HTTP/1.1',
            'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
            'accept-encoding': 'gzip, deflate, sdch',
            'accept-language': 'ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4',
            'cache-control': 'no-cache',
            'pragma': 'no-cache',
            'upgrade-insecure-requests': '1',
            'user-agent': 'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36',
            'x-compress': 'null',
        })
        browser = RoboBrowser(session=session, history=True, parser='lxml')

        browser.open(work_url)

        phone_link = browser.find('a', {"class": "action-show-number"}).attrs['href'] + '?async'
        browser.session.headers[':path'] = phone_link
        browser.session.headers['accept'] = 'application/json, text/javascript, */*; q=0.01'
        browser.session.headers['referer'] = work_url
        browser.session.headers['upgrade-insecure-requests'] = ''
        browser.session.headers['x-requested-with'] = 'XMLHttpRequest'
        phone = browser.open(self.main_url + phone_link)

        print(self.main_url + phone_link, phone)
        print(browser.session.headers)
        print(browser.session.cookies)

if __name__ == '__main__':
    settings = { 'main_url': 'https://m.avito.ru', 'output_file': 'output.csv' }
    aggregator = Aggregator(settings)
    aggregator.start_process()
@ljluestc
Copy link

from selenium import webdriver
import time

# Set up the Selenium WebDriver (you may need to install the appropriate driver for your browser)
driver = webdriver.Chrome()

# Open the initial URL
initial_url = "https://m.avito.ru/sankt-peterburg/predlozheniya_uslug/almaznoe_burenie_almaznaya_rezka_usilenie_79225740"
driver.get(initial_url)

# Wait for the page to load completely (you may need to adjust the wait time)
time.sleep(5)

# Find the phone link and click it
phone_link = driver.find_element_by_css_selector(".action-show-number")
phone_link.click()

# Wait for the AJAX request to complete (you may need to adjust the wait time)
time.sleep(5)

# Extract the phone number
phone_number = driver.find_element_by_css_selector(".js-item-phone-number").text
print("Phone number:", phone_number)

# Close the browser
driver.quit()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants