Integrating selenium/not submitting form? #19

jmsp · 2018-06-26T19:37:01Z

I am using Selenium/Python to try and fill out a form and than fill out the recaptcha. I found python-anticaptcha and bought $10 in credits, and everything is working, the captcha comes up, but than nothing happens. I tried to look for answers for a few hours/consulted their api and examples, but could not find anything. Ultimately, the captcha should work and then the website would generate a table which I am trying to web-scrape

This is what it ends up looking like, but nothing happens and after a minute or so it usually quits, this is the code

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
from python_anticaptcha import AnticaptchaClient, NoCaptchaTaskProxylessTask
import re
import pandas as pd
import os
import time
import requests

url = "https://claimittexas.org/app/claim-search"
driver = webdriver.Safari()
driver.implicitly_wait(30)
driver.get(url)

wait = WebDriverWait(driver, 30)
result = driver.find_element_by_xpath('//*[@id="lastName"]')
driver.execute_script("arguments[0].value='Al';",result)
time.sleep(2)
result.submit()

api_key = '**REDACTED BY @ad-m**'
site_key = '6LeQLyEUAAAAAKTwLC-xVC0wGDFIqPg1q3Ofam5M'  # grab from site

time.sleep(2)
client = AnticaptchaClient(api_key)
task = NoCaptchaTaskProxylessTask(url, site_key)
job = client.createTask(task)
job.join()
token = job.get_solution_response()
requests.post(url, data={'g-recaptcha-response': token}).text

The text was updated successfully, but these errors were encountered:

ad-m · 2018-06-26T20:57:04Z

@jmsp , using the requests and selenium at the same time does not make sense. See #10 how to integrate selenium & python-anticaptcha. If the given example is not enough for you - let me know and I will develop a solution for your specific case.

jmsp · 2018-06-26T22:35:25Z

@ad-m I just tried that, and it says this - Traceback (most recent call last):

  File "texas.py", line 44, in <module>
    driver.find_element_by_xpath('//input[@type="image"]').click()
  File "/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py
", line 387, in find_element_by_xpath
    return self.find_element(by=By.XPATH, value=xpath)
  File "/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py
", line 957, in find_element
    'value': value})['value']
  File "/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py
", line 314, in execute
    self.error_handler.check_response(response)
  File "/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler
.py", line 242, in check_response

I used this code


from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
from python_anticaptcha import AnticaptchaClient, NoCaptchaTaskProxylessTask
import re
import pandas as pd
import os
import time
import requests

url = "https://claimittexas.org/app/claim-search"
driver = webdriver.Safari()
driver.implicitly_wait(30)
driver.get(url)

wait = WebDriverWait(driver, 30)
result = driver.find_element_by_xpath('//*[@id="lastName"]')
driver.execute_script("arguments[0].value='Al';",result)
time.sleep(2)
result.submit()

api_key = '**REDACTED BY @ad-m**'
site_key = '6LeQLyEUAAAAAKTwLC-xVC0wGDFIqPg1q3Ofam5M'  # grab from site

time.sleep(2)
client = AnticaptchaClient(api_key)
task = NoCaptchaTaskProxylessTask(url, site_key)
job = client.createTask(task)
job.join()
# Receive response
response = job.get_solution_response()

# Inject response in webpage
driver.execute_script('document.getElementById("g-recaptcha-response").innerHTML = "%s"' % (response))

# Wait a moment to execute the script (just in case).
time.sleep(1)

# Press submit button
driver.find_element_by_xpath('//input[@type="image"]').click()

ad-m · 2018-06-26T22:37:30Z

I need 12 hours to looks over that. Now I am going to sleep. See you soon.

ad-m · 2018-06-27T13:04:02Z

@jmsp , here is working example for required website. In works on my local station:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
from python_anticaptcha import AnticaptchaClient, NoCaptchaTaskProxylessTask
import re
import pandas as pd
import os
import time
import requests

url = "https://claimittexas.org/app/claim-search"
driver = webdriver.Chrome()
driver.implicitly_wait(30)
driver.get(url)

lastNameField = driver.find_element_by_xpath('//input[@id="lastName"]')
lastNameField.send_keys('Al')

api_key = '....'
site_key = '6LeQLyEUAAAAAKTwLC-xVC0wGDFIqPg1q3Ofam5M'  # grab from site

client = AnticaptchaClient(api_key)
task = NoCaptchaTaskProxylessTask(url, site_key)
job = client.createTask(task)
print("Waiting to solution by Anticaptcha workers")
job.join()
# Receive response
response = job.get_solution_response()
print("Received solution", response)

# Inject response in webpage
driver.execute_script('document.getElementById("g-recaptcha-response").innerHTML = "%s"' % response)

# Wait a moment to execute the script (just in case).
time.sleep(1)

# Press submit button
driver.find_element_by_xpath('//button[@type="submit" and @class="btn-std"]').click()

jmsp · 2018-06-27T17:44:20Z

Does this work with safari or I have to use chromedriver? it didn't work for me when I switched it to safari

ad-m · 2018-06-27T18:17:38Z

I don't have Safari, so I don't have way to test it. If exists any problem to use Safari in Selenium - this is Safari/Selenium problem, nor python-anticaptcha.

ad-m · 2018-06-27T22:47:21Z

Do you need additional support? Can you check on Chrome? Can you close the application?

ad-m · 2018-06-30T10:17:22Z

Is there anything I can help you with?

ad-m · 2018-07-17T21:02:57Z

I do not see any need for further help, so I am goint to close that issue. If you need further support - let me know.

ad-m · 2019-06-10T09:01:23Z

For anybody which may it concer I added constatly-tested example how to use python_anticaptcha and selenium together: https://github.com/ad-m/python-anticaptcha/blob/master/examples/recaptcha_selenium.py

MacMarde · 2019-06-25T01:03:07Z

@ad-m
I really tried hard using your code with selenium and chrome.
I want to login on https://www.swagbucks.com/p/login

I can't get it working. I am getting following error message:

in process token = get_token(url, site_key, invisible_captcha) 
in get_token is_invisible=invisible
TypeError: __init__() got an unexpected keyword argument 'is_invisible'

is_invisible is set to True globally. Changing it to False or removing it does also not work.
Please help me. I really have no idea what I am doing wrong.

MacMarde · 2019-06-25T23:34:15Z

It is working if I am removing is_invisible like this:

def get_token(url, site_key):
    task = NoCaptchaTaskProxylessTask(
        website_url=url,
        website_key=site_key
    )
    job = client.createTask(task)
    job.join()
return job.get_solution_response()

But now I am getting another error from:

 form_submit
 driver.execute_script("onSuccess('{}')".format(token))

with the error:
selenium.common.exceptions.JavascriptException: Message: javascript error: onSuccess is not defined`

Any ideas?

I also tried the following:

driver.execute_script('document.getElementById("g-recaptcha-response").innerHTML = "%s"' % token)
time.sleep(1)
driver.find_element_by_xpath('//button[@type="submit" and @class="btn-std"]').click()

But I again got an error code saying:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[@type="submit" and @class="btn-std"]"}

I tried using XPathHelper to get the XPath from the submit button, but XPathHelper won't work with this button.

I have tried this solution:

driver.execute_script('document.getElementById("g-recaptcha-response").innerHTML = "%s"' % token)
    time.sleep(1)
    frame1 = driver.find_element_by_xpath(".//iframe[@title='recaptcha challenge']")
    driver.switch_to.frame(frame1)
    driver.find_element_by_xpath(".//*[@id='recaptcha-verify-button']").click()

With this method the recaptcha submit button gets clicked, but then the recaptcha frame just says "Please select all matching images." So the recapthca is not solved.

MacMarde · 2019-06-26T01:43:57Z

I am also not sure if I do have the correct site-key.
I cant find an entry like:
<div class="g-recaptcha" data-sitekey="6Lc_aCMTAAAAABx7u2W0WPXnVbI_v6ZdbM6rYf16"></div>

I also cant use firebug to find it, because firebug is no longer available. With Firefox Developer its also not working because I never get a google captcha. And firebug with chrome is also not working because I cant see the traffic. Why is there no applicable solution documented?

I can only find this with chrome Developer Tools:

var welcomeVid = "//www.youtube.com/embed/0z96q2Aixig?rel=0&autoplay=1&wmode=opaque";
            sbGlbl.loginLp = sbGlbl.loginLp();
            sbGlbl.loginLp.onResizeRunner();
            $(window).bind('resize', sbGlbl.loginLp.onResizeRunner);
            sbGlbl_IsLoginLanding = true;
            sbGlbl.captchaRequired = true;
            sbGlbl.captchaSitekey = '6Ld48JYUAAAAAGBYDutKlRp2ggwiDzfl1iApfaxE';

This is the best I can get. I hope its the correct site key.

ivanchenko2022 · 2022-04-13T01:36:04Z

Hello

ad-m closed this as completed Jul 17, 2018

ad-m mentioned this issue Sep 13, 2018

How to deal with task with no form submission and use callbacks #21

Closed

MacMarde mentioned this issue Jun 26, 2019

Again Integrating selenium/not submitting form #37

Closed

ad-m added the selenium label Jun 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrating selenium/not submitting form? #19

Integrating selenium/not submitting form? #19

jmsp commented Jun 26, 2018 •

edited by ad-m

ad-m commented Jun 26, 2018

jmsp commented Jun 26, 2018 •

edited by ad-m

ad-m commented Jun 26, 2018

ad-m commented Jun 27, 2018 •

edited

jmsp commented Jun 27, 2018

ad-m commented Jun 27, 2018

ad-m commented Jun 27, 2018

ad-m commented Jun 30, 2018

ad-m commented Jul 17, 2018

ad-m commented Jun 10, 2019

MacMarde commented Jun 25, 2019

MacMarde commented Jun 25, 2019 •

edited

MacMarde commented Jun 26, 2019 •

edited

ivanchenko2022 commented Apr 13, 2022

Integrating selenium/not submitting form? #19

Integrating selenium/not submitting form? #19

Comments

jmsp commented Jun 26, 2018 • edited by ad-m

ad-m commented Jun 26, 2018

jmsp commented Jun 26, 2018 • edited by ad-m

ad-m commented Jun 26, 2018

ad-m commented Jun 27, 2018 • edited

jmsp commented Jun 27, 2018

ad-m commented Jun 27, 2018

ad-m commented Jun 27, 2018

ad-m commented Jun 30, 2018

ad-m commented Jul 17, 2018

ad-m commented Jun 10, 2019

MacMarde commented Jun 25, 2019

MacMarde commented Jun 25, 2019 • edited

MacMarde commented Jun 26, 2019 • edited

ivanchenko2022 commented Apr 13, 2022

jmsp commented Jun 26, 2018 •

edited by ad-m

jmsp commented Jun 26, 2018 •

edited by ad-m

ad-m commented Jun 27, 2018 •

edited

MacMarde commented Jun 25, 2019 •

edited

MacMarde commented Jun 26, 2019 •

edited