New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement #196 Add file downloading mechanism without "href" #267

Closed
wants to merge 2 commits into
from

Conversation

Projects
None yet
3 participants
@ddemin
Contributor

ddemin commented Jan 9, 2016

@asolntsev please review first version of implementation #196 issue.

How to use in end-user projects:

        Selenide.prepareDownload("C:\\SomeFolder", "", "application/x-sdlc");
        WebDriverRunner.getWebDriver().get("http://javadl.sun.com/webapps/download/AutoDL?BundleId=113227");
        // Downloaded file with name chromeinstall-8u66.exe
        File tmp = new File(Selenide.$(By.tagName("body")).getText());

or

        Selenide.prepareDownload("C:\\SomeFolder","SomeFile.exe","application/x-sdlc");
        WebDriverRunner.getWebDriver().get("http://javadl.sun.com/webapps/download/AutoDL?BundleId=113227");
        // Downloaded file with name SomeFile.exe
        File tmp = new File(Selenide.$(By.tagName("body")).getText());

or

        Selenide.$(By.id("button_for_file_download_after_click"))
                .prepareDownload("C:\\SomeFolder", "", "application/pdf")
                .click();
        File tmp = new File(Selenide.$(By.tagName("body")).getText());

TODO:

  1. Review
  2. Some tests
  3. Merge conflicts
@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Jan 10, 2016

Contributor

Fix

Contributor

ddemin commented Jan 10, 2016

Fix

@asolntsev

This comment has been minimized.

Show comment
Hide comment
@asolntsev

asolntsev Jan 10, 2016

Member

@dimand58 Thank you! Great! It's almost what we need.

The only problem is the API. The new downloading mechanism is particularly needed in situation when we don't know exact URL. You typically fill some form with 100500 fields, click "submit", and file starts downloading.

Try for example:

  • http://idemo.bspb.ru/#corporate
  • Cards -> Acquiring -> "View terminals"
  • Put some checkboxes and click "Report"
  • Expected result: an XLS file starts downloading

I suggest to make existing method $.download() more powerful without adding any new methods.

This method:

  File report = $("button").download();

currently only works with elements <a href=***>.
What I suggest is to make it work with any downloads. It could work like this:

  1. Test calls $("button").download();
  2. Selenide sets some ThreadLocal flag like Selenide.browserMobProxyActive.set(true);
  3. Selenide clicks the button
  4. Because of flag, BMP stores all received files to some folder.
  5. Selenide method download returns first of these files.
  6. Selenide method download resets flag back: Selenide.browserMobProxyActive.set(false);
  7. While this flag is false, BMP does not store any files (to avoid degrading performance).
Member

asolntsev commented Jan 10, 2016

@dimand58 Thank you! Great! It's almost what we need.

The only problem is the API. The new downloading mechanism is particularly needed in situation when we don't know exact URL. You typically fill some form with 100500 fields, click "submit", and file starts downloading.

Try for example:

  • http://idemo.bspb.ru/#corporate
  • Cards -> Acquiring -> "View terminals"
  • Put some checkboxes and click "Report"
  • Expected result: an XLS file starts downloading

I suggest to make existing method $.download() more powerful without adding any new methods.

This method:

  File report = $("button").download();

currently only works with elements <a href=***>.
What I suggest is to make it work with any downloads. It could work like this:

  1. Test calls $("button").download();
  2. Selenide sets some ThreadLocal flag like Selenide.browserMobProxyActive.set(true);
  3. Selenide clicks the button
  4. Because of flag, BMP stores all received files to some folder.
  5. Selenide method download returns first of these files.
  6. Selenide method download resets flag back: Selenide.browserMobProxyActive.set(false);
  7. While this flag is false, BMP does not store any files (to avoid degrading performance).
@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Jan 17, 2016

Contributor

@asolntsev 2 problem exists:

  1. BrowserMob must be activated BEFORE of WebDriver instance
  2. Web must know ALL mime types for files

Because of this problems my implementation looks more appropriative.

TODO:

  • add support of BMob configuration before initiating (chained proxy and etc)
Contributor

ddemin commented Jan 17, 2016

@asolntsev 2 problem exists:

  1. BrowserMob must be activated BEFORE of WebDriver instance
  2. Web must know ALL mime types for files

Because of this problems my implementation looks more appropriative.

TODO:

  • add support of BMob configuration before initiating (chained proxy and etc)
@asolntsev

This comment has been minimized.

Show comment
Hide comment
@asolntsev

asolntsev Jan 18, 2016

Member

@dimand58

  1. BrowserMob must be activated BEFORE of WebDriver instance
    Yes, but it's not a problem. Selenide will activate BrowserMob in the very beginning (before creating webdriver). But this BrowserMob instance will not save any files. It will just proxy requests from webdriver to server. Then, method $.download() will set some internal flag to TRUE, and only then BrowserMob will save all responses with header like "attachment" to local files.

  2. Web must know ALL mime types for files
    I suggest that Selenide can return the first saved file with header like "attachment".

With your implementation, use needs to know the exact URL for downloading file. But user cannot always know it. For example, file can be downloaded as a result of form submission.

Member

asolntsev commented Jan 18, 2016

@dimand58

  1. BrowserMob must be activated BEFORE of WebDriver instance
    Yes, but it's not a problem. Selenide will activate BrowserMob in the very beginning (before creating webdriver). But this BrowserMob instance will not save any files. It will just proxy requests from webdriver to server. Then, method $.download() will set some internal flag to TRUE, and only then BrowserMob will save all responses with header like "attachment" to local files.

  2. Web must know ALL mime types for files
    I suggest that Selenide can return the first saved file with header like "attachment".

With your implementation, use needs to know the exact URL for downloading file. But user cannot always know it. For example, file can be downloaded as a result of form submission.

@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Jan 19, 2016

Contributor

OK.
Response filter will be save file if HTTP response header have "Content-Disposition: attachment;". In this case no need to know MIME type of downloaded file.

@asolntsev But what about redirections?
Test case:

  1. Go to http://www.java.com/ru/download/chrome.jsp
  2. Click on BIG RED BUTTON (java dowload)

Actual:

  1. No Content-Disposition in response header
  2. 1 redirect occured (to http://sdlc-esd.oracle.com/ESD6/JSCDL/jdk/8u66-b18/chromeinstall-8u66.exe)
  3. AGAIN - no Content-Disposition in response header

I think Selenide must have option to catch files by Content-Type in response header too.

Contributor

ddemin commented Jan 19, 2016

OK.
Response filter will be save file if HTTP response header have "Content-Disposition: attachment;". In this case no need to know MIME type of downloaded file.

@asolntsev But what about redirections?
Test case:

  1. Go to http://www.java.com/ru/download/chrome.jsp
  2. Click on BIG RED BUTTON (java dowload)

Actual:

  1. No Content-Disposition in response header
  2. 1 redirect occured (to http://sdlc-esd.oracle.com/ESD6/JSCDL/jdk/8u66-b18/chromeinstall-8u66.exe)
  3. AGAIN - no Content-Disposition in response header

I think Selenide must have option to catch files by Content-Type in response header too.

@mplushnikov

This comment has been minimized.

Show comment
Hide comment
@mplushnikov

mplushnikov Mar 2, 2016

+1 for downloading files from elements without href attribute, like buttons.

+1 for downloading files from elements without href attribute, like buttons.

@asolntsev

This comment has been minimized.

Show comment
Hide comment
@asolntsev

asolntsev Mar 2, 2016

Member

@mplushnikov agree. We need this. I will work on it soon.

Member

asolntsev commented Mar 2, 2016

@mplushnikov agree. We need this. I will work on it soon.

@asolntsev asolntsev self-assigned this Apr 3, 2016

@asolntsev asolntsev added this to the 3.6 milestone Apr 3, 2016

@asolntsev

This comment has been minimized.

Show comment
Hide comment
@asolntsev

asolntsev Apr 20, 2016

Member

@dimand58 Sorry for the long delay. I am almost ready to merge the pull request, but I have one problem.

BMP doesn't work for me when I run tests locally (against http://127.0.0.1:9000). But it works for remote sites like google.com. See some comments here: http://automated-testing.info/t/browsermob-proxy-java-webdriver-pomogite-zapustit-prostejshij-test/4531/32

Don't you know a solution?

Member

asolntsev commented Apr 20, 2016

@dimand58 Sorry for the long delay. I am almost ready to merge the pull request, but I have one problem.

BMP doesn't work for me when I run tests locally (against http://127.0.0.1:9000). But it works for remote sites like google.com. See some comments here: http://automated-testing.info/t/browsermob-proxy-java-webdriver-pomogite-zapustit-prostejshij-test/4531/32

Don't you know a solution?

@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Apr 20, 2016

Contributor

@asolntsev , hi!
Give me a few days, I will try to solve this problem.

Contributor

ddemin commented Apr 20, 2016

@asolntsev , hi!
Give me a few days, I will try to solve this problem.

@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Apr 20, 2016

Contributor

@asolntsev
Are you tried to use DNS name of your PC instead of 127.0.0.1 ? I suppose that this trick can help

Contributor

ddemin commented Apr 20, 2016

@asolntsev
Are you tried to use DNS name of your PC instead of 127.0.0.1 ? I suppose that this trick can help

@asolntsev

This comment has been minimized.

Show comment
Hide comment
@asolntsev

asolntsev Apr 22, 2016

Member

How can I use name of computer? I cannot hardcode it in tests - tests
should be easily runnable on any machine.
On Apr 21, 2016 1:25 AM, "Demin Dmitrii" notifications@github.com wrote:

@asolntsev https://github.com/asolntsev
Are you tried to use DNS name of your PC instead of 127.0.0.1 ? I suppose
that this trick can help


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#267 (comment)

Member

asolntsev commented Apr 22, 2016

How can I use name of computer? I cannot hardcode it in tests - tests
should be easily runnable on any machine.
On Apr 21, 2016 1:25 AM, "Demin Dmitrii" notifications@github.com wrote:

@asolntsev https://github.com/asolntsev
Are you tried to use DNS name of your PC instead of 127.0.0.1 ? I suppose
that this trick can help


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#267 (comment)

@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Apr 22, 2016

Contributor

You can get current pc name in code, via some method maybe

Contributor

ddemin commented Apr 22, 2016

You can get current pc name in code, via some method maybe

@asolntsev

This comment has been minimized.

Show comment
Hide comment
@asolntsev

asolntsev Apr 22, 2016

Member

Yes, something like InternetAddress.getLocal().getHost() should return name
of current computer. But it can be very slow, depending on network
configuration.

Why couldn't BMP just work with IP? Both Jetty an Netty can.
On Apr 22, 2016 12:03 PM, "Demin Dmitrii" notifications@github.com wrote:

You can get current pc name in code, via some method maybe


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#267 (comment)

Member

asolntsev commented Apr 22, 2016

Yes, something like InternetAddress.getLocal().getHost() should return name
of current computer. But it can be very slow, depending on network
configuration.

Why couldn't BMP just work with IP? Both Jetty an Netty can.
On Apr 22, 2016 12:03 PM, "Demin Dmitrii" notifications@github.com wrote:

You can get current pc name in code, via some method maybe


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#267 (comment)

@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Apr 22, 2016

Contributor

@asolntsev What does happen when your test opens http://127.0.0.1:9000 via proxy?
I tried on my PC (opened http://127.0.0.1/ Denwer URL) and all works great.

Contributor

ddemin commented Apr 22, 2016

@asolntsev What does happen when your test opens http://127.0.0.1:9000 via proxy?
I tried on my PC (opened http://127.0.0.1/ Denwer URL) and all works great.

@asolntsev

This comment has been minimized.

Show comment
Hide comment
@asolntsev

asolntsev Apr 22, 2016

Member

This is my case that does not work:

  1. open http://127.0.0.1:9000 via proxy
  2. download a file

Result: proxy intercepts all requests, but does not intercept server
response. So I cannot get the downloaded file.
Expected result: I want to get server response.

Andrei Solntsev

2016-04-22 19:03 GMT+03:00 Demin Dmitrii notifications@github.com:

@asolntsev https://github.com/asolntsev What does happen when your test
opens http://127.0.0.1:9000 via proxy?
I tried on my PC and all works great.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#267 (comment)

Member

asolntsev commented Apr 22, 2016

This is my case that does not work:

  1. open http://127.0.0.1:9000 via proxy
  2. download a file

Result: proxy intercepts all requests, but does not intercept server
response. So I cannot get the downloaded file.
Expected result: I want to get server response.

Andrei Solntsev

2016-04-22 19:03 GMT+03:00 Demin Dmitrii notifications@github.com:

@asolntsev https://github.com/asolntsev What does happen when your test
opens http://127.0.0.1:9000 via proxy?
I tried on my PC and all works great.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#267 (comment)

@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Apr 22, 2016

Contributor

@asolntsev it is strange...
My steps:

  1. Init BMP before first WD creation
BrowserMobProxy proxy = new BrowserMobProxyServer();
 {
     proxy.start(1234);
     Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
     WebDriverRunner.setProxy(seleniumProxy);
 }
  1. Add filter before URL opening:
    proxy.addResponseFilter(new ResponseFilterImpl());
  2. Open URL:
    Selenide.open("http://127.0.0.1");
  3. In debug mode I can see call of filterResponse method in ResponseFilterImpl
Contributor

ddemin commented Apr 22, 2016

@asolntsev it is strange...
My steps:

  1. Init BMP before first WD creation
BrowserMobProxy proxy = new BrowserMobProxyServer();
 {
     proxy.start(1234);
     Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
     WebDriverRunner.setProxy(seleniumProxy);
 }
  1. Add filter before URL opening:
    proxy.addResponseFilter(new ResponseFilterImpl());
  2. Open URL:
    Selenide.open("http://127.0.0.1");
  3. In debug mode I can see call of filterResponse method in ResponseFilterImpl
@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Apr 22, 2016

Contributor

Browser - Google Chrome
BMP:

<dependency>
        <groupId>net.lightbody.bmp</groupId>
        <artifactId>browsermob-core-littleproxy</artifactId>
        <version>2.1.0-beta-5</version>
</dependency>

Selenide 3.4

Contributor

ddemin commented Apr 22, 2016

Browser - Google Chrome
BMP:

<dependency>
        <groupId>net.lightbody.bmp</groupId>
        <artifactId>browsermob-core-littleproxy</artifactId>
        <version>2.1.0-beta-5</version>
</dependency>

Selenide 3.4

@asolntsev

This comment has been minimized.

Show comment
Hide comment
@asolntsev

asolntsev Apr 25, 2016

Member

@dimand58 Please review my failing test. In Selenide master branch, there are 2 tests:

  • integration.proxy.BrowserMobProxyTest - it uses deprecated net.lightbody.bmp.proxy.ProxyServer - and it works
  • integration.proxy.BrowserMobProxyServerTest - it uses newer net.lightbody.bmp.BrowserMobProxyServer - and it doesn't work for localhost. Please take a look at test local_https() that is currently ignored.

@dimand58 Can you say why this test fails?

Member

asolntsev commented Apr 25, 2016

@dimand58 Please review my failing test. In Selenide master branch, there are 2 tests:

  • integration.proxy.BrowserMobProxyTest - it uses deprecated net.lightbody.bmp.proxy.ProxyServer - and it works
  • integration.proxy.BrowserMobProxyServerTest - it uses newer net.lightbody.bmp.BrowserMobProxyServer - and it doesn't work for localhost. Please take a look at test local_https() that is currently ignored.

@dimand58 Can you say why this test fails?

@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Apr 25, 2016

Contributor

@asolntsev , got it. I will investigate this test at evening today

Contributor

ddemin commented Apr 25, 2016

@asolntsev , got it. I will investigate this test at evening today

@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Apr 25, 2016

Contributor

@asolntsev can you update your certificate? It can be one of reason of current problem.
image

I had similar problem on one of my project and it was solved via fixing of SSL certificate

Contributor

ddemin commented Apr 25, 2016

@asolntsev can you update your certificate? It can be one of reason of current problem.
image

I had similar problem on one of my project and it was solved via fixing of SSL certificate

@asolntsev

This comment has been minimized.

Show comment
Hide comment
@asolntsev

asolntsev Apr 25, 2016

Member

Do you mean the SSL certificate of https://localhost:xxxx?

I deliberately use self-signed (untrusted) SSL certificate there because I
want to test how Selenide works with untrusted certs. It's because many
companies use untrusted certs on their test-servers.

Typically webdrivers have method like "setTrustCerts(true)", I guess BMP
also has such a method.

Andrei Solntsev

2016-04-25 17:57 GMT+03:00 Dmitriy Demin notifications@github.com:

@asolntsev https://github.com/asolntsev can you update your
certificate? It can be one of reason of current problem.
[image: image]
https://cloud.githubusercontent.com/assets/5477035/14788108/284bdfaa-0b0f-11e6-9f10-61bd22aa66a3.png


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#267 (comment)

Member

asolntsev commented Apr 25, 2016

Do you mean the SSL certificate of https://localhost:xxxx?

I deliberately use self-signed (untrusted) SSL certificate there because I
want to test how Selenide works with untrusted certs. It's because many
companies use untrusted certs on their test-servers.

Typically webdrivers have method like "setTrustCerts(true)", I guess BMP
also has such a method.

Andrei Solntsev

2016-04-25 17:57 GMT+03:00 Dmitriy Demin notifications@github.com:

@asolntsev https://github.com/asolntsev can you update your
certificate? It can be one of reason of current problem.
[image: image]
https://cloud.githubusercontent.com/assets/5477035/14788108/284bdfaa-0b0f-11e6-9f10-61bd22aa66a3.png


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#267 (comment)

@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin Apr 25, 2016

Contributor

I created new self-signed cert (with valid expiration date and from/to fields 127.0.0.1) but problem still exists.
However, in IE11 page opens via proxy! I'm totaly confused

Contributor

ddemin commented Apr 25, 2016

I created new self-signed cert (with valid expiration date and from/to fields 127.0.0.1) but problem still exists.
However, in IE11 page opens via proxy! I'm totaly confused

@ddemin

This comment has been minimized.

Show comment
Hide comment
@ddemin

ddemin May 10, 2016

Contributor

@asolntsev
I've found and checked the solution - we should use latest version of BPM (beta-7-SNAPSHOT which not realised yet). With that BMP version local web works good.
I guess we should to wait a little bit

Contributor

ddemin commented May 10, 2016

@asolntsev
I've found and checked the solution - we should use latest version of BPM (beta-7-SNAPSHOT which not realised yet). With that BMP version local web works good.
I guess we should to wait a little bit

@asolntsev asolntsev modified the milestones: 3.6, 3.9 Sep 19, 2016

@asolntsev

This comment has been minimized.

Show comment
Hide comment
@asolntsev

asolntsev Sep 19, 2016

Member

@dimand58 I implemented this idea a little bit differently in Selenide 3.9

Huge thanks for your ideas and support!

Member

asolntsev commented Sep 19, 2016

@dimand58 I implemented this idea a little bit differently in Selenide 3.9

Huge thanks for your ideas and support!

@asolntsev asolntsev closed this Sep 19, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment