[🚀 Feature]: Add File Download Support to Bindings

[titusfortner]

Feature and motivation

In #11277 there is now support for moving files from a Grid standalone/node to the client machine.

A flag has to be set when the grid is started, but we should provide a Selenium specific endpoint and method for getting that from inside the bindings themselves.

@asolntsev I see you're tracking it for Selenide already, want to help get it in Selenium? selenide/selenide#1687

Usage example

String base64 = driver.download("my-file-name.pdf");

Or:

Zip remoteFile = driver.download("my-file-name.pdf");

Or Maybe:

driver.transferFromGrid("my-file-name.pdf", "/location/to/save/file");

[github-actions]

This issue is looking for contributors.

Please comment below or reach out to us through our IRC/Slack/Matrix channels if you are interested.

[asolntsev]

@titusfortner Sure, with a great pleasure. :)

But I have few suggestions:

1. We would need an endpoint for getting a list of all downloaded files. In many cases, we don't know the exact file name, and need to detect the downloaded file by comparing file list before and after user action.
2. Will it be new method only in RemoteWebDriver class?
3. I would not name it download. Rather getDownloadedFile() (and listDownloadedFiles()).

[titusfortner]

For the list I think we would need a separate implementation in the grid, right @krmahadevan?

[krmahadevan]

@titusfortner

So it looks like we have the below asks:

1. List the files that were downloaded till now.
2. Given a file name, retrieve its contents (This is present already)
3. Given a list of files, retrieve their contents.

The GET URL that currently responds to (2) is /session/:sessionId:/se/file?filename=

{
   "filename":"input.txt",
   "contents": "base64EncodedStringThatRepresentsAZippedFileContentsGoesHere"
}

To support (1), we could alter the behaviour such that if the user does a GET on /session/:sessionId:/se/file we can return back a payload that looks like

[
	{
		"filename":"input.txt",
		"contents": ""
	},
	{
		"filename":"anotherFile.txt",
		"contents": ""
	}
]

To support (3) we can have the user specify a comma separated list of file names /session/:sessionId:/se/file?filename=file1.txt,file2.txt

which would return back a payload of

[
	{
		"filename":"file1.txt",
		"contents": "base64EncodedStringThatRepresentsAZippedFileContentsGoesHere"
	},
	{
		"filename":"file2.txt",
		"contents": "base64EncodedStringThatRepresentsAZippedFileContentsGoesHere"
	}
]

So we wouldn't need to add additional URLs but can work within whatever we have already. Its just that for (1) wherein user asks for the list of files that can be downloaded it would be kind of awkward to send back a response but without the contents. But if we have a client API then we could perhaps abstract out this.

The client binding could either reside within the RemoteWebDriver (or) maybe we could do something like

DownloadManager dm = new DownloadManager(driver);
driver.transferFromGridAndUnzip("my-file-name.pdf", "/location/to/save/file"); //Since we give back only a zip file

Please let me know what do you think

[jsa34]

Just a quick comment about waits - I haven't checked the current implementation, but in order to avoid querying for a file that hasn't yet downloaded, should we create a wait or build the wait into fetch of the file (retry with polling intervals and timeout)?

[diemol]

List the files that were downloaded till now.

This makes sense. However, I would put this behind a flag because some users might be using a shared Grid and they just do not want to allow listing the directory contents. Also, it gets tricky because what if more than one session has downloaded a file? Should we list all files no matter the session that downloaded it or should we list only the files that were downloaded in that particular session? This all implies that we could potentially need to manage a state of downloaded files in the Node? I'd really like to avoid such overhead in the Node code (that already handles quite a bit).

Given a list of files, retrieve their contents.

I do not see the need for this. What is the use case that is not covered by a single file download? I would not expect users now using the Grid to download large amounts of file per session.

Just a quick comment about waits - I haven't checked the current implementation, but in order to avoid querying for a file that hasn't yet downloaded, should we create a wait or build the wait into fetch of the file (retry with polling intervals and timeout)?

This is the trickiest one because the download happens in a remote machine. AFAIK, there are only workarounds to know if a file has been completely downloaded or not, nothing really standard. That is why our suggestion is to alter the system under test to offer a way of knowing that the file has been downloaded.

Another thing I forgot to mention in the initial implementation is that we forgot to add some mechanism to clean downloaded files, otherwise the Node machine can run out of space at some point.

[krmahadevan]

@diemol

What is the use case that is not covered by a single file download? I would not expect users now using the Grid to download large amounts of file per session.

Lets say that a test basically downloads a file multiple times. Since a test cannot specify the name of the file to be saved as, depending on the OS, the file name gets chosen. For e.g., in OSX I noticed that, if there's a file name already available in the Downloads folder, then OSX appends ( 1 ), ( 2 ) and so on to the file name. So in this case, the user would perhaps not know what the actual file name is going to be. Another use case can be wherein a web application generates a file on the file and then saves it. In this case also the file name would not be available.
So maybe in such situations, it would be useful for a person to know the files that are available for download.

Maybe we can restrict the files on a session id basis (wherein inside the downloads path that a user specifies the respective files would be downloaded under a folder that represents a session id). That way, for a given session a user would be able to retrieve his/her downloaded files, but that list would perhaps never span across sessions and so the ask of My downloaded files should not be visible to the other user when I am sharing the same grid can be addressed.

we forgot to add some mechanism to clean downloaded files, otherwise the Node machine can run out of space at some point.

Should we drive it via a flag wherein we say if set to autoClean as true, then we basically clean up the folder every-time the node comes up ?

[krmahadevan]

Just a quick comment about waits - I haven't checked the current implementation, but in order to avoid querying for a file that hasn't yet downloaded, should we create a wait or build the wait into fetch of the file (retry with polling intervals and timeout)?

In the normal mode without the grid in the picture, today how do you determine if a file has been downloaded?

• Do you wait for some predetermined duration (or)
• Do you poll a directory wherein the file is going to be saved to check if its available ?

[diemol]

Lets say that a test basically downloads a file multiple times. Since a test cannot specify the name of the file to be saved as, depending on the OS, the file name gets chosen. For e.g., in OSX I noticed that, if there's a file name already available in the Downloads folder, then OSX appends ( 1 ), ( 2 ) and so on to the file name. So in this case, the user would perhaps not know what the actual file name is going to be. Another use case can be wherein a web application generates a file on the file and then saves it. In this case also the file name would not be available.
So maybe in such situations, it would be useful for a person to know the files that are available for download.

This is covered by allowing the user to list the downloaded files, and then choosing one of them.

Maybe we can restrict the files on a session id basis (wherein inside the downloads path that a user specifies the respective files would be downloaded under a folder that represents a session id). That way, for a given session a user would be able to retrieve his/her downloaded files, but that list would perhaps never span across sessions and so the ask of My downloaded files should not be visible to the other user when I am sharing the same grid can be addressed.

AFAIK, this is not possible because the browser already has a downloads directory and the file download process is between the browser and the host OS, WebDriver is not involved in the process of determining the directory or the file name.

we forgot to add some mechanism to clean downloaded files, otherwise the Node machine can run out of space at some point.
Should we drive it via a flag wherein we say if set to autoClean as true, then we basically clean up the folder every-time the node comes up ?

This should be handled in a different issue, I think.

[titusfortner]

I agree with @diemol wrt not needing the "get all" feature, user can iterate over the list and download whatever they want in subsequent commands.

If we don't want scope to go crazy here, we should also refer to this as a beta feature.

put this behind a flag

Simplest option is you opt into the whole downloads feature with --downloads-path. If that isn't set, the grid should return an error, if it is set, list and get should both work.

should we list only the files that were downloaded in that particular session

Current implementation won't differentiate between something downloaded during the session vs already there... I think if we start the grid with --downloads-path /path/to/dir/goes/here we should put the downloaded files in /path/to/dir/goes/here/<sessionid>/. @diemol if we're adding this as a special thing in the bindings, I think we need to create a method in the applicable Options classes, and if we do that, we can add the session id as well. Makes sense that the end user needs to opt-in to this with a specific Options method as well.

some mechanism to clean downloaded files

I think driver quit / session end should clear everything in directory with that session id.

there are only workarounds to know if a file has been completely downloaded or not

I'm not an expert, don't browsers download files to a temp name and then rename it when complete for just this reason? Or is that just a chrome thing? Either way, waits are out of scope, imo. Up to the user to figure out what and how to wait.

[titusfortner]

Also I'm not sure we're following the spec to the extent we need to.

I wasn't following the original PR, just saw when it was merged. Can we do this without the query string? Is that valid w3c syntax, even for a custom endpoint?

Right now we have this for uploading a file to remote:

POST session/:session_id/se/file {"file": <zipped_file>}

We can do this like we do cookies:

GET /session/:sessionId:/se/file
GET /session/:sessionId:/se/file/file-name.png

I'm not sure we can even put file name like that, so probably want to do it like we do elements:

POST /session/:sessionId:/se/file  {"name": file-name}

Obviously return syntax needs to put payload inside "value" object, just like all other commands.

[titusfortner]

Actually... Why does the grid need to start with the parameter. Couldn't grid access whatever was passed into the capabilities if we make the end user pass it in the capabilities?

[diemol]

Current implementation won't differentiate between something downloaded during the session vs already there... I think if we start the grid with --downloads-path /path/to/dir/goes/here we should put the downloaded files in /path/to/dir/goes/here//. @diemol if we're adding this as a special thing in the bindings, I think we need to create a method in the applicable Options classes, and if we do that, we can add the session id as well. Makes sense that the end user needs to opt-in to this with a specific Options method as well.

How can you tell the browser to download the file to a directory with the session as a prefix?

Actually... Why does the grid need to start with the parameter. Couldn't grid access whatever was passed into the capabilities if we make the end user pass it in the capabilities?

It could be a security hole, which would allow a client to access any file on the remote machine.

[asolntsev]

In the normal mode without the grid in the picture, today how do you determine if a file has been downloaded?
Yes, that's the topic I wanted to talk about on the following SeleniumConf: https://confengine.com/conferences/selenium-conference-chicago-2023/proposal/17439/why-its-so-hard-to-download-a-file

There are several options, none of which is perfect. There is lots of hidden tricks and underwater rocks. All browsers do it differently (even chromium-based).

@titusfortner Let's keep it very simple:

1. Every new browser in Grid creates a new (unique!) folder for downloading files.
2. It doesn't even need to be configurable. It's the only default behaviour.
3. Method "get downloaded files" returns all files from this unique directory. It's not any security problem.
4. No waiting is needed. Just return current state of the folder.

To implement #1, there are options in most browsers:

1. Chromium-based: preferences.put("download.default_directory", browserDownloadsFolder.getAbsolutePath());
2. Firefox: firefoxOptions.addPreference("browser.download.dir", browserDownloadsFolder.getAbsolutePath());
3. IE: downloads folder is not customizable
4. Safari: downloads folder is not customizable

[titusfortner]

How can you tell the browser to download the file to a directory with the session as a prefix?

Oh, shoot, yeah, you don't know the session id, yet ☹️
I guess it doesn't have to be the session ID directly, it can be any uuid. But would look like:

GET /session/:sessionId:/se/file/:uuid:
POST /session/:sessionId:/se/file/:uuid:  {"name": file-name}

Can the grid map an id to the session to delete the directory at the end? Or is that especially complicated?

It could be a security hole

Oh, yeah, can we restrict it to HOME? Theoretically sensitive things don't get stored in HOME? But probably smart to require grid to "enable" the feature somehow. Just don't like both the grid config & end user to have to have the same String for it to work... Can we just hard code a specific directory? Follow Selenium Manager template?
$HOME/.cache/selenium/downloads

java -jar selenium-server.jar --enable-downloads

@asolntsev I think we're going the same direction, except the grid shouldn't be changing the options passed in. I think the bindings need to have a separate method that enables it, and can set the full directory with unique identifier; then the grid can parse it to create the directory.

ChromeOptions options = new ChromeOptions();
options.enableRemoteDownloads();

public void enableRemoteDownloads() {
  setBinary(getHomeDirectory() + ".cache/selenium/downloads/" + generateId())
}

[diemol]

Keep in mind that the Node is just passing commands along, it does not know the file name either. When a file is downloaded, it will be placed in the downloads directory and all that concerns the Grid is that a command was executed, it does not even know how to wait for the file to be downloaded. Any attempt to figure that out will be a workaround that needs to be maintained.

Yeah, it'd be nice to map the session or any given id to the downloaded file and delete all that when the session is over. But again, the Node does not know the file name to keep that mapping. Plus, that would make the Node more complex that it is already.

I think the current approach of having a flag with the directory is good enough because it is an opt in, and it gives the user the possibility to define which directory is the one that will be used (of course, given that the browser uses that as well).

I would add the endpoints we want, and let people tell us if this is good enough or not.

[krmahadevan]

We can do this like we do cookies:

GET /session/:sessionId:/se/file
GET /session/:sessionId:/se/file/file-name.png

I'm not sure we can even put file name like that, so probably want to do it like we do elements:

POST /session/:sessionId:/se/file  {"name": file-name}

Obviously return syntax needs to put payload inside "value" object, just like all other commands.

@titusfortner

Should this be a separate bug that tracks this ask or is this refactoring also being captured in this same feature request? Also what should the endpoint be ? GET or POST ?

[krmahadevan]

@titusfortner @diemol

I agree with @diemol wrt not needing the "get all" feature, user can iterate over the list and download whatever they want in subsequent commands.

This currently does not exist. So does it mean that one of the feature requirements would be, to be able to list all files in a directory.

Is it still the downloads path that was specified at the time of starting the hub or

Based on what @asolntsev is suggesting, is this a unique sub-directory under the downloads-path parent directory that the client bindings will pass via the Options is a separate ask. Also is this something that the Grid is going to inject on its own or is the client bindings expected to add this ?

@asolntsev I think we're going the same direction, except the grid shouldn't be changing the options passed in. I think the bindings need to have a separate method that enables it, and can set the full directory with unique identifier; then the grid can parse it to create the directory.

@titusfortner - Wouldn't this now require a test to be able to know the directory structure of an actual node? It needs to know the OS flavour as well to be able to specify a full directory path because the folder needs to be adherant to the OS specifics [ like its C:\foo\bar in windows and its /Users/foo/bar in NON Windows ].

@diemol @titusfortner - Just thinking out loud here. What if we did something like this ?

1. When the session match happens and a new session is created at the node and when the downloads-path is specified at the node level, then we :
   a. Add the session id as the downloads path to the capabilities object (if it's not present). If it's present, then maybe we throw an exception stating that it's now an illegal state to specify a downloads directory when the node has already been instructed to download files to a specific feature?
   b. Create a directory within the downloads path whose name would be the session id.
2. Now when a user requests for list of files or the file name, we would be able to retrieve the files from the session id sub directory under the downloads path.

Some questions that crop up:

1. From @asolntsev explanation, I understand that both IE and Safari don't support the notion of allowing a user to specify the downloads directory. So in those cases, what should the Grid's response be, when the User tries to invoke this API to fetch a file ? Should we tell the user that its not supported or should we just try to query from the directory that the user specified and if the file is not found (which would be true when this directory does not match with the defaults download directory) we just behave as if its a normal file not found scenario ?
2. Since the downloads directory is a setting at the node level, there's always a chance that there may be 1 or more nodes in the Grid farm for which this setting is not made available. Now this is a setting that the client is not expected to know. So when the client uses the new API that we are looking at building for letting a user retrieve a downloaded file from the node, and when that API ends up getting routed to a node that does not have this path specified, what should the user experience be ? Should it throw an error or should it pretend as if the File was NOT found ?

[diemol]

This currently does not exist. So does it mean that one of the feature requirements would be, to be able to list all files in a directory.
Is it still the downloads path that was specified at the time of starting the hub or
Based on what @asolntsev is suggesting, is this a unique sub-directory under the downloads-path parent directory that the client bindings will pass via the Options is a separate ask. Also is this something that the Grid is going to inject on its own or is the client bindings expected to add this ?

With the current implementation, the directory needs to exist. Plus, that idea of manipulating options from the bindings is complicated because that needs to be set at the beginning of the session, and you do not know if the user wants to download a file during the session. Seems there will be a lot of magic involved, which I do not want to have in the code.

This issue is now growing with comments about different features. I will open separate issues for each one:

• Clean up downloaded files - [🚀 Feature]: Cleanup downloaded files in Node (Grid) #11457
• List all downloaded files - [🚀 Feature]: List all downloaded files in a session (Grid) #11458

[titusfortner]

@krmahadevan GET vs POST is just a matter of whether the info you are requesting can be put in the endpoint itself, or has to be specified in a payload.

[titusfortner]

Problem

We could implement everything as-is in the bindings. User hard codes a location in both node setup and in options, and everything goes to the same directory.

1. There is no reason to designate a particular directory; from a user perspective either you want to get your downloads or you don't, you shouldn't need to know which directory they were stored in
2. It's annoying/burdensome to have to put the same string in 2 different, non-obvious places
3. All downloads from all sessions go to the same place, which *will cause endless headaches

Easiest Partial Solution

The 3rd issue is obviously the biggest, and we could potentially address it by adding a UUID subdirectory in the bindings.

• User starts a node with a --download-directory <path>
• User creates options instance and callssetDownloadPath(<path>)
• This method generates a uuid and adds <path>/<uuid> to the proper prefs
• The node would have to create that subdirectory
• If node can associate the uuid with the session id [edit: actually, does node have access to the capabilities at all parts of the session? would be really easy to add the file path to an se:downloadsPath capability], then it can use it when file endpoints are called & delete the directory at end session
• If node can't do this, bindings could store it in driver instance and pass it along with the endpoints so node doesn't need to store anything; Can also implement DELETE endpoint that bindings can send as part of quit() methods.

Any reason why this wouldn't work?

Full Solution Ideas

Ideally we could handle everything for the user, the trickiest part is knowing the architecture of node if user isn't providing the path.

• user starts node with --enable-downloads
• user creates options instance and calls enableDownloads()
• Everything just works for listing and downloading 😄

Questions:

1. Can we rely on platformName to inform us the location of $HOME directory? (can node specify the platformName it responds to as whatever it wants?)
2. Does hub know anything more about the end node architecture other than what is in the capabilities? The node knows, which is what actually matters, but the node can't adjust browser vendor capabilities, so it doesn't matter.

If the answer to both of those is "no" I think we can probably actually rely on Selenium Manager to solve this for us.
The bindings can set a "se:enableDownloads": true capability from the options enableDownloads() method.
We could get the Selenium Manager to check for that capability. It knows the correct path to $HOME/.cache/selenium and will soon be able to access/modify the options. It could create a unique subdirectory for the session and set the download location in the options instance before it gets sent to the driver (could do this in both browser specific location and in an se namespaced capability so node can track it). Then it's just up to the grid to be able to store the location for deletion at the end?

[krmahadevan]

@titusfortner

• User specifies the download sub-path (for e.g., onboarding) via the Options at the time of creating the WebDriver.
• At the node end, the moment we see that the download sub-path is specified in the capabilities inside the method org.openqa.selenium.grid.node.local.LocalNode#newSession, we go ahead and do the following:
  • Was the node to which the session got routed to, started with downloads-path. If not, then fail the session creation ?
  • If the path has been set, then after session gets created, create sub folder under downloads-path provided via capability sub-path.

Not clear for me.

• The notion of session id being one of the sub-folders under downloads-path/sub-path (or) under downloads-path is not clear, because the actual downloading of a file via the browser is not something that the node can intercept on and thus ensure that it lands up exactly under a session-id sub folder.

• How to ensure that the download-path that is set at the time of starting the node, does not get shared amongst users (in a grid setup) ? I was thinking that this should be enforced by the Grid user, by adding a custom capability at the time of booting the node, and that same custom capability should be matched when creating a new webdriver instance.

Once I get clarity on this part, I will proceed with working on adding changes and raise a PR.

[titusfortner]

Thinking through this again after a month away... I think the grid node is powerful enough to do everything we need without the user having to do much.

User shouldn't set location of downloads for security reasons. We create a unique directory associated with a session and that's what the user gets.

I think this should work:

1. User starts a node with --enable-downloads (or adds it as a required capability so that it can be matched per step 4)
2. User calls enableDownloads(true) on Options instance
3. Bindings take that method and add "se:enableDownloads": true to capabilities
4. Grid Hub parses that capability and makes sure a node exists with that flag and errors if one isn't found
5. Node parses that capability and does:
   • Creates UUID (can't be session ID, because grid needs to know the download location before the session ID is created)
   • Creates directory $HOME/.cache/selenium/downloads/<UUID>
   • Adds that directory to the applicable vendor-namespaced capabilities as appropriate
   • Stores that location for use with the various /file or /files endpoints (this is the part where I'm not sure how Node manages state/configs)

[krmahadevan]

@titusfortner - I know we are going back and forth on this, but how do we deal with the situation of actually controlling where the downloads happen?

I remember @diemol mentioned this earlier as well, WebDriver does not have any control in the actual file download, and so we cannot even determine in which directory was the file downloaded.

For some browsers, I believe that can be controlled by the user as called out earlier by @asolntsev in this comment

This is what confuses me:

We can control the entire download folder management part of it, but we really don't have any say in where exactly does the file get downloaded.

So how do we move forward.

If we had a Webdriver way of controlling the downloads (similar to how we do the upload of a file) then we would have been in a better place I feel.

If we go by the approach that you have cited, then we should at-least send back the download directory to the user via the capabilities object and then maybe have an api at the RemoteWebDriver (or) within the Options classes, which can give this information to the user, so that they can use it to set the downloads location for at-least firefox, chrome and edge. Is that acceptable ?

[krmahadevan]

If we go by the approach that you have cited, then we should at-least send back the download directory to the user via the capabilities object and then maybe have an api at the RemoteWebDriver (or) within the Options classes, which can give this information to the user, so that they can use it to set the downloads location for at-least firefox, chrome and edge. Is that acceptable ?

Now that I think of it, even this doesn't make sense, because I think for chromium and firefox browsers we would be setting this at the time of webdriver instantiation via the Options class. So this kind of becomes like a chicken and egg problem :(

[titusfortner]

how do we deal with the situation of actually controlling where the downloads happen?

The node should be able to do everything without the user needing to send or receive anything specific. The node creates the directory and modifies the vendor capabilities before it passes it in to the driver. The capabilities that need to be modified will be different for each browser (and won't be possible for some browsers).

[krmahadevan]

@titusfortner @diemol

(Based on the conversation that I have had with @titusfortner on this) expectations from this feature are as below:

• This applies only for browsers that do support the notion of letting a user specify a download location.
• When a user would like the grid to manage the downloads for them, they specify the intent via a new custom capability named "se:enableDownloads": true when a RemoteWebDriver object is being created.
• For this feature to work there should be at-least one Grid node that has been started with a new command line flag named --enable-downloads
• The Grid's capability matching feature should ensure that it tries to match the capability which contains "se:enableDownloads": true ONLY with those nodes that have been booted with --enable-downloads
• If no match was found, we fall back to the same logic that we currently use to reject a session request when capabilities don't match.
• Now as part of honouring the new session, the grid creates a new sub directory under the directory specified via --downloads-dir (This will be like the base directory where all downloads go through and which will be periodically cleaned up either at boot time or shutdown time) and then remembers this directory for the newly created session (session id to sub directory mapping to be maintained somewhere)
• After a user performs downloads and then does /files, we list all the files available for download for the specific session from the user (We should try to keep files to be downloaded on a per session basis)
• When a user specifies the file to download via /file we download that specific file from the session mapped sub directory.

I will work on getting this done and come back in case i have any specific doubts.

[titusfortner]

Yes! Except a couple more suggestions.

The node should create a UUID to associate with the session to keep items from different sessions in different subdirectories.

the grid creates a new sub directory under the directory specified via --downloads-dir

I don't think we need to let the user change this directory. I think we should follow what Selenium Manager is doing and put it in $HOME/.cache/selenium directory.

[diemol]

Maybe we can break this into smaller PRs? Like, one that changes the current endpoint to avoid using the query string (#11466). Then another one to process the capability and create the directory associated to the session, and from there we continue.

[titusfortner]

@diemol Breaking this up into the following and closing this one:

1. Change current implementation of download to be w3c compliant — [🐛 Bug]: Grid File download capability should be w3c compliant  #11466
2. Implement getting list of downloads — [🚀 Feature]: List all downloaded files in a session (Grid) #11458
3. Implement node features to support downloads with capability — [🚀 Feature]: Grid implement capability to set download directory #11656
4. Implement bindings support for grid features — [🚀 Feature]: Add Download Support to Bindings #11657
5. Ensure capabilities or settings for downloads are matched — [🚀 Feature]: Grid properly matches enable download support #11658