Add file downloading mechanism without "href" #196

Closed
asolntsev opened this Issue Jul 12, 2015 · 25 comments

Projects

None yet

4 participants

@asolntsev
Member

Sometimes file download starts without clicking link with "href" attribute. Selenide could provide way to download those files too.

I guess it's only possible to do using proxy server (like browsermob-proxy).

@asolntsev asolntsev added the feature label Jul 12, 2015
@leslyarun

@asolntsev Can u pls provide me with an example of using browsermob-proxy in Selenide ???

@leslyarun

@asolntsev I'm using a corporate proxy. How to make browsermob to use the same proxy server ? please provide me with an example

@asolntsev
Member

I don't know exactly.
I guess BrowserMobProxy could be able to intercept all http requests and
forward them to your corporate proxy. But I don't know if it actually can
do this.

If it cannot, we need to find or create such a proxy server.

Andrei Solntsev

2015-08-05 14:31 GMT+03:00 leslyarun notifications@github.com:

@asolntsev https://github.com/asolntsev I'm using a corporate proxy.
How to make browsermob to use the same proxy server ? please provide me
with an example


Reply to this email directly or view it on GitHub
#196 (comment).

@leslyarun

I'm using a proxy address and port no. for accessing the internet. But when I use BrowserMob, it creates it's own proxy server.
If there is an way to make browsermob to use a particular proxy address and port no. , it would be very helpful. Do u have any idea ? @asolntsev

@asolntsev
Member

@leslyarun I don't know exactly if there is a way. This requires some investigation. If you get it sooner, please inform me.

@dimand58
Contributor
dimand58 commented Jan 4, 2016

Hi all. It's simple!
You can tune Browser Mob for auto downloading of files based on Content-Type header.
Test case:

  1. Navigate to some URL
  2. Click on button which lead to response with Content-Type = application/pdf (downloading of PDF file after click)
  3. Browser Mob Proxy catch Content-Type=application/pdf, save file to temp folder and replace response body with full path to PDF file

First of all you must add&start BMob (You can do it in @BeforeSuite in TestNG as example):

        // proxyServer  - field in class with type BrowserMobProxyServer
        proxyServer = new BrowserMobProxyServer(port);
        proxyServer.start();
        WebDriverRunner.setProxy(ClientUtil.createSeleniumProxy(proxyServer));

Then you must to create response filter (thanks @barancev)

public class FileDownloader implements ResponseFilter {

    private Set<String> contentTypes = new HashSet<String>();
    // ExConfiguration.CONFIG_DEFAULT_DOWNLOAD_PATH - my own field with absolute path to temporary folder
    private File tempDir = new File(ExConfiguration.CONFIG_DEFAULT_DOWNLOAD_PATH);
    private File tempFile = null;

    public FileDownloader addContentType(String contentType) {
        contentTypes.add(contentType);
        return this;
    }


    public static FileDownloader withContent(String contentType)
    {
        return new FileDownloader().addContentType(contentType);
    }


    public static FileDownloader withContents(String ... contentType)
    {
        FileDownloader downloader = new FileDownloader();
        for (int i=0; i < contentType.length; i++)
        {
            downloader.addContentType(contentType[i]);
        }
        return downloader;
    }


    @Override
    public void filterResponse(io.netty.handler.codec.http.HttpResponse response, HttpMessageContents contents, HttpMessageInfo messageInfo) {
        String contentType = response.headers().get("Content-Type");
        if (contentTypes.contains(contentType)) {
            try {
                String postfix = contentType.substring(contentType.indexOf('/') + 1);
                tempFile = File.createTempFile("downloaded", "." + postfix, tempDir);
                tempFile.deleteOnExit();

                FileOutputStream outputStream = new FileOutputStream(tempFile);

                outputStream.write(contents.getBinaryContents());
                outputStream.close();

                response.headers().remove("Content-Type");
                response.headers().remove("Content-Encoding");
                response.headers().remove("Content-Disposition");

                response.headers().add("Content-Type", "text/html");
                response.headers().add("Content-Length", "" + tempFile.getAbsolutePath().length());
                contents.setTextContents(tempFile.getAbsolutePath());
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

then set-up content in your test (PDF as example):

    @Test
    public void someTest() {
        proxyServer.addResponseFilter(FileDownloader.withContent("application/pdf"));
        Selenide.open("http://someurl.com");
        $(By.id("btn_for_start_pdfdownloading_without_href")).click();

        File downloadedPdf = new File($(By.tagName("body")).getText()));
        // You can use downloadedPdf as you wish
    }

Do not forget to stop BMob (in @AfterSuite as example)

        proxyServer.stop();
@dimand58
Contributor
dimand58 commented Jan 4, 2016

@asolntsev can you add this into Selenide doc ?

@asolntsev
Member

@dimand58 yes, I am going to add this functionality to Selenide, so that user will not need to copy-paste this code.

@dimand58
Contributor
dimand58 commented Jan 5, 2016

@asolntsev Will be great! Can I help with it?

@asolntsev
Member

@dimand58 Sure! Pull request would be very helpful.
P.S. It seems that BrowserMobProxy doesn't work with latest Jetty versions. So, if I use latest Jetty in my project, I cannot use BrowserMobProxy :(
I am right? Is there any solution?

@asolntsev asolntsev added the lets do it label Jan 7, 2016
@dimand58
Contributor
dimand58 commented Jan 7, 2016

@asolntsev I will investigate it

@dimand58
Contributor
dimand58 commented Jan 7, 2016

@asolntsev Try with

      <dependency>
           <groupId>net.lightbody.bmp</groupId>
           <artifactId>browsermob-core-littleproxy</artifactId>
           <version>2.1.0-beta-4</version>
       </dependency>

It works for me (with Selenide 3.0)

@dimand58
Contributor
dimand58 commented Jan 7, 2016

@asolntsev

Sure! Pull request would be very helpful.

I have one problem - I don't know what kind of solution will be most appropriate and clear. Can you provide your 'high-level' vision of using BMob in Selenide?

@dimand58
Contributor
dimand58 commented Jan 9, 2016

@asolntsev please investigate PR #267

@asolntsev
Member

I have described my vision in comments for PR #267

@asolntsev asolntsev added a commit that referenced this issue Aug 21, 2016
@asolntsev asolntsev #196 fix the test 1ae0ab8
@asolntsev asolntsev added a commit that referenced this issue Aug 21, 2016
@asolntsev asolntsev #196 fix the test 0f8c145
@asolntsev asolntsev added a commit that referenced this issue Aug 21, 2016
@asolntsev asolntsev #196 fix the test 9d2bdd2
@asolntsev asolntsev added this to the 3.9.1 milestone Aug 26, 2016
@asolntsev asolntsev added a commit that referenced this issue Aug 26, 2016
@asolntsev asolntsev #196 "abort" proxy server instead of "stopping" it
"abort" does not wait for completion of all current requests
26df904
@asolntsev asolntsev added a commit that referenced this issue Aug 26, 2016
@asolntsev asolntsev #196 shutdown proxy server at the same moment (in the same thread) wh…
…en browser is closed
2a54a2f
@asolntsev
Member

Selenide 3.9.1 will be release in few hours. It will implement downloading files via BrowserMobProxy.

@dimand58 Huge thanks for your contribution to this functionality!!!

@asolntsev asolntsev closed this Aug 27, 2016
@asolntsev
Member

Now Selenide method $.download() can download any files using proxy server.

How it works:

  1. Test executes command $("selector").download();
  2. Selenide activates proxy-server
  3. Selenide clicks the element
  4. Proxy server tracks all server responses that contain http header Content-Disposition.
    Proxy server extracts file name from this header, and saves such a file in folder build/reports/tests.
  5. Selenide waits (up to 4 seconds) until some file(s) is downloaded.
  6. Method $.download() returns the first downloaded file.
  7. If some new windows have been opened at this time, Selenide closes all of them.
    It's needed when PDF is opened in a new browser window (inline).
@thasherwin

Wow. That's awesome

@thasherwin

I tested the new version and currently I don’t have a way to make download work with my own capabilities setup connected to a VPN with proxy server.

  1. Is there a way to do this with Selenide using your own capabilities when you are connected to VPN with a proxy server?
  2. What is the best practice to create a Selenide webdriver instance together with Selenium grid? Currently i do this by creating my own capabilities and tell selenide to use my webdriver instance.

Below a few scenarios that I used to test the download functionality:
Scenario 1:
Test download with no VPN connection and proxy
Code:

@Test
public void testIfFileCanBeDownloadedWithOutVPNConnection(){
    open(url);
    try {
            File downloadedFile = $(".btn.btn-default").download();
            assertThat("Check if file: mailmerge.xls is downloaded", "mailmerge.xls", org.hamcrest.Matchers.equalTo(downloadedFile.getName()));

    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }
}

Actual:
It works!!

Scenario 2:
Test download with VPN connection and proxy configured using Selenide to create the webdriver instance

@Test
public void testIfFileCanBeDownloadededOnVPNWithProxy(){
    Proxy p=new Proxy();
    p.setHttpProxy("xxxx.local:8080");
    WebDriverRunner.setProxy(p);
    open (url);
    try {
        File downloadedFile = $(".btn.btn-default").download();
        assertThat("Check if file: mailmerge.xls is downloaded", "mailmerge.xls", org.hamcrest.Matchers.equalTo(downloadedFile.getName()));
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }
}

Actual:
It works!!

Scenario 3:
Test download with my own capabilities together with a proxy server and connected to a VPN using Selenide.

@Test
public void testIfFileCanBeDownloadedWithVPNConnectionAndProxyAndOwnCap(){
    DesiredCapabilities cap = DesiredCapabilities.firefox();
    cap.setBrowserName("firefox");
    Proxy p=new Proxy();
    p.setHttpProxy("xxxx.local:8080");
    WebDriverRunner.setWebDriver(new FirefoxDriver(cap));
    WebDriverRunner.setProxy(p);
    open (url);
    try {
            File downloadedFile = $(".btn.btn-default").download();
            assertThat("Check if file: mailmerge.xls is downloaded", "mailmerge.xls", org.hamcrest.Matchers.equalTo(downloadedFile.getName()));
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }
}

Actual:
it does not work.
Page does not load.
Probably my code is wrong. I am not configuring the proxy properly

Scenario 4:
Test selenide with my own capabilities setup together Selenide connected to VPN. I did not mention any proxy. I guess i do need to mention the proxy to prevent an exception

@Test
public void testIfFileCanBeDownloadedWithVPNCOnnectionAndOwnCap(){
    DesiredCapabilities cap = DesiredCapabilities.firefox();
    cap.setBrowserName("firefox");
    Proxy p=new Proxy();
    p.setHttpProxy("xxxx.local:8080");
    WebDriverRunner.setWebDriver(new FirefoxDriver(cap));
    WebDriverRunner.setProxy(p);
    open (url);
    try {
            File downloadedFile = $(".btn.btn-default").download();
            assertThat("Check if file: mailmerge.xls is downloaded", "mailmerge.xls", org.hamcrest.Matchers.equalTo(downloadedFile.getName()));
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }
}

Actual:
It does not work.
Page does load but I get the following exception. I guess it's related to the proxy.

java.lang.NullPointerException
    at com.codeborne.selenide.commands.DownloadFile.clickAndInterceptFileByProxyServer(DownloadFile.java:46)
    at com.codeborne.selenide.commands.DownloadFile.execute(DownloadFile.java:37)
    at com.codeborne.selenide.commands.DownloadFile.execute(DownloadFile.java:25)
    at com.codeborne.selenide.commands.Commands.execute(Commands.java:137)
    at com.codeborne.selenide.impl.SelenideElementProxy.dispatchAndRetry(SelenideElementProxy.java:85)
    at com.codeborne.selenide.impl.SelenideElementProxy.invoke(SelenideElementProxy.java:61)
    at com.sun.proxy.$Proxy5.download(Unknown Source)
    at DownloadFile.testDownload.testIfFileCanBeDownloadedWithVPNCOnnectionx(testDownload.java:97)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:86)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:643)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1128)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
    at org.testng.TestRunner.privateRun(TestRunner.java:782)
    at org.testng.TestRunner.run(TestRunner.java:632)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
    at org.testng.SuiteRunner.run(SuiteRunner.java:268)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
    at org.testng.TestNG.run(TestNG.java:1064)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:74)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:124)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

@asolntsev
Member

@thasherwin Wow! Thank you for such a detailed report.

Yes, currently Selenide does download files through its own proxy server.
If you create a browser instance by yourself, Selenide cannot add own proxy to it. It's not technically possible.

What we could do is provide public method like WebDriverRunner.getSelenideProxy(), so that everyone who creates his own browser can use Selenide proxy for it.

But are you sure you really need to create browser by yourself? In both Scenario 3 and Scenario 4 it's not really needed - you just create the same browser as Selenide would create automatically.

PS. This line of your code has wrong order of arguments:
assertThat("Check if file: mailmerge.xls is downloaded", "mailmerge.xls", equalTo(downloadedFile.getName()));

The correct order is:
assertThat("Check if file: mailmerge.xls is downloaded", downloadedFile.getName(), equalTo("mailmerge.xls"));

And actually the message is not really useful, so that I would just leave short line:
assertThat(downloadedFile.getName(), equalTo("mailmerge.xls"));

@thasherwin

@asolntsev
Thanks for the quick reply.
Well the only reason now to create my own webdriver instance is for Selenium grid. Please see the below example.

        WebDriver driver = null;
        DesiredCapabilities capability = DesiredCapabilities.firefox();
        capability.setBrowserName("firefox");
        capability.setPlatform(Platform.WINDOWS);
        capability.setCapability("machineName", "Open_Machine_FF");
        try {
            driver = new RemoteWebDriver(new URL("http://test.local:4444/wd/hub"), capability);
            WebDriverRunner.setWebDriver(driver);
        }catch(Exception e){

        }

Is it possible to create a Selenide RemoteWebDriver instance for selenium grid? If this is possible, than there is not a big need for me to have my own browser incance.

Also thanks for your comments on my assertion. I will update my code :)

@asolntsev
Member

it's easy. You just need to set both properties:
Configuration.browser = "chrome"; // or "firefox"
Configuration.remote = "http://localhost:4444/wd/hub";

It's enough. Selenide will create remote webdriver with needed capabilities.
(Well, I am not sure about "machineName" capability. Can you try?)

Andrei Solntsev

2016-08-28 11:26 GMT+03:00 thasherwin notifications@github.com:

@asolntsev https://github.com/asolntsev
Thanks for the quick reply.
Well the only reason now to create my own webdriver instance is for
Selenium grid. Please see the below example.

    WebDriver driver = null;
    DesiredCapabilities capability = DesiredCapabilities.firefox();
    capability.setBrowserName("firefox");
    capability.setPlatform(Platform.WINDOWS);
    capability.setCapability("machineName", "Open_Machine_FF");
    try {
        driver = new RemoteWebDriver(new URL("http://test.local:4444/wd/hub"), capability);
        WebDriverRunner.setWebDriver(driver);
    }catch(Exception e){

    }

Is it possible to create a Selenide RemoteWebDriver instance for selenium
grid? If this is possible, than there is not a big need for me to have my
own browser incance.

Also thanks for your comments on my assertion. I will update my code :)


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#196 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARE3VjddbnOTb9h9TKCWPREcGKOEjbIks5qkUYcgaJpZM4FXEwI
.

@thasherwin

@asolntsev
I tested it on the grid using;

 Configuration.browser = "chrome"; // or "firefox"
  Configuration.remote = "http://localhost:4444/wd/hub";

and it works. I can now close my issue #317 related to downloads using jenkins :)
I do mis the capabilty option. With that i can tell selenium to use a specific machine connected the grid. Maybe something for a pull request if this option is not yet built?

Thanks

@asolntsev
Member

Yes, definitely. It's a perfect candidate for pull request. Let's do it!

On Aug 29, 2016 11:59 AM, "thasherwin" notifications@github.com wrote:

@asolntsev https://github.com/asolntsev
I tested it on the grid using;

Configuration.browser = "chrome"; // or "firefox"
Configuration.remote = "http://localhost:4444/wd/hub";

and it works. I can now close my issue #317
#317 related to downloads
using jenkins :)
I do mis the capabilty option. With that i can tell selenium to use a
specific machine connected the grid. Maybe something for a pull request if
this option is not yet built?

Thanks


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#196 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARE3WXYiSWkyd93MnxjLWnFaA7PYOZ6ks5qkp9ZgaJpZM4FXEwI
.

@asolntsev
Member

Yes, definitely. It's a perfect candidate for pull request. Let's do it!

On Aug 29, 2016 11:59 AM, "thasherwin" notifications@github.com wrote:

@asolntsev https://github.com/asolntsev
I tested it on the grid using;

Configuration.browser = "chrome"; // or "firefox"
Configuration.remote = "http://localhost:4444/wd/hub";

and it works. I can now close my issue #317
#317 related to downloads
using jenkins :)
I do mis the capabilty option. With that i can tell selenium to use a
specific machine connected the grid. Maybe something for a pull request if
this option is not yet built?

Thanks


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#196 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARE3WXYiSWkyd93MnxjLWnFaA7PYOZ6ks5qkp9ZgaJpZM4FXEwI
.

@BorisOsipov BorisOsipov added a commit to BorisOsipov/selenide that referenced this issue Nov 23, 2016
@asolntsev @BorisOsipov asolntsev + BorisOsipov #196 upgrade to browsermob-core 2.1.2 - it can work with http://local…
…host
01ebcee
@BorisOsipov BorisOsipov added a commit to BorisOsipov/selenide that referenced this issue Nov 23, 2016
@asolntsev @BorisOsipov asolntsev + BorisOsipov #196 clear uploaded files before every test fe7e267
@BorisOsipov BorisOsipov added a commit to BorisOsipov/selenide that referenced this issue Nov 23, 2016
@asolntsev @BorisOsipov asolntsev + BorisOsipov #196 fix the test 7f66ee3
@BorisOsipov BorisOsipov added a commit to BorisOsipov/selenide that referenced this issue Nov 23, 2016
@asolntsev @BorisOsipov asolntsev + BorisOsipov #196 fix the test 9d85ebd
@BorisOsipov BorisOsipov added a commit to BorisOsipov/selenide that referenced this issue Nov 23, 2016
@asolntsev @BorisOsipov asolntsev + BorisOsipov #196 fix the test c1bfd0d
@BorisOsipov BorisOsipov added a commit to BorisOsipov/selenide that referenced this issue Nov 23, 2016
@asolntsev @BorisOsipov asolntsev + BorisOsipov #196 ignore 2 failing tests in phantomjs 90f0624
@BorisOsipov BorisOsipov added a commit to BorisOsipov/selenide that referenced this issue Nov 23, 2016
@asolntsev @BorisOsipov asolntsev + BorisOsipov #196 "abort" proxy server instead of "stopping" it
"abort" does not wait for completion of all current requests
db3c3e2
@BorisOsipov BorisOsipov added a commit to BorisOsipov/selenide that referenced this issue Nov 23, 2016
@asolntsev @BorisOsipov asolntsev + BorisOsipov #196 shutdown proxy server at the same moment (in the same thread) wh…
…en browser is closed
b1c8c34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment