Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Uploads fail after server to server redirect #1980

Open
2 tasks
JustinKyleJames opened this issue Jun 21, 2021 · 2 comments
Open
2 tasks

Uploads fail after server to server redirect #1980

JustinKyleJames opened this issue Jun 21, 2021 · 2 comments
Assignees
@JustinKyleJames
Labels
consortium-member enhancement

Comments

@JustinKyleJames
Copy link
Contributor

JustinKyleJames commented Jun 21, 2021 •
edited
Loading

  • master
  • 4-2-stable

The 4.2.9.0 plugin fails on uploads when there is a server redirect because the necessary information (file size, #threads, #client requested threads, and oprType) are not being made available to the new server.

Temporary workaround (4.2.9.1):

  • Force cache on redirect
  • Use a file open counter (counts up and down) to determine when the cache file is ready to be flushed rather than a simply countdown starting from the #threads.

Note on the second bullet, when we are streaming we would sometimes get one thread call open after the others have called both open/closed which fooled the plugin into thinking the processing was done and complete multipart would be called too soon. This doesn't appear to be an issue when cache is used. (This is either because of timing differences or some recent server changes fixed this.)

Permanent solution (probably 4.2.10.0, b/c change needed in the server core, not just this plugin):

  • Make sure the necessary data is being sent on redirect so streaming works.
@JustinKyleJames JustinKyleJames changed the title Uploads file after server to server redirect Uploads fail after server to server redirect Jun 21, 2021
JustinKyleJames added a commit to JustinKyleJames/irods_resource_plugin_s3 that referenced this issue Jun 23, 2021
@JustinKyleJames
[irods#1980] Fix server upload after server redirect.
aa215fe 
For now, on redirect the plugin can't get file size, #threads, requested #threads,
or oprType.  These are all needed.

This will force the use of a cache file.  Since we don't know the # threads, I added
a file open counter and flush the cache file when that goes down to zero.

Note:  Previously, I had noticed that one thread may have a delayed open/write/close
which made using a file open counter problematic as all others may close before this
thread opened.  That does not appear to be happening now - at least under this
scenario.
JustinKyleJames added a commit to JustinKyleJames/irods_resource_plugin_s3 that referenced this issue Jun 23, 2021
@JustinKyleJames
[irods#1980] Fix server upload after server redirect.
739d554 
For now, on redirect the plugin can't get file size, #threads, requested #threads,
or oprType.  These are all needed.

This will force the use of a cache file.  Since we don't know the # threads, I added
a file open counter and flush the cache file when that goes down to zero.

Note:  Previously, I had noticed that one thread may have a delayed open/write/close
which made using a file open counter problematic as all others may close before this
thread opened.  That does not appear to be happening now - at least under this
scenario.
trel pushed a commit that referenced this issue Jun 24, 2021
@JustinKyleJames @trel
[#1980] Fix server upload after server redirect.
3b47f9f 
For now, on redirect the plugin can't get file size, #threads, requested #threads,
or oprType.  These are all needed.

This will force the use of a cache file.  Since we don't know the # threads, I added
a file open counter and flush the cache file when that goes down to zero.

Note:  Previously, I had noticed that one thread may have a delayed open/write/close
which made using a file open counter problematic as all others may close before this
thread opened.  That does not appear to be happening now - at least under this
scenario.
@JustinKyleJames
Copy link
Contributor Author

The workaround has been implemented in 4-2-stable. I will not check mark it until the permanent solution is implemented.

JustinKyleJames added a commit to JustinKyleJames/irods_resource_plugin_s3 that referenced this issue Nov 8, 2021
@JustinKyleJames
[irods#1980] Fix server upload after server redirect.
f8168a3 
For now, on redirect the plugin can't get file size, #threads, requested #threads,
or oprType.  These are all needed.

This will force the use of a cache file.  Since we don't know the # threads, I added
a file open counter and flush the cache file when that goes down to zero.

Note:  Previously, I had noticed that one thread may have a delayed open/write/close
which made using a file open counter problematic as all others may close before this
thread opened.  That does not appear to be happening now - at least under this
scenario.
JustinKyleJames added a commit to JustinKyleJames/irods_resource_plugin_s3 that referenced this issue Nov 11, 2021
@JustinKyleJames
[irods#1980] Fix server upload after server redirect.
c9151d3 
For now, on redirect the plugin can't get file size, #threads, requested #threads,
or oprType.  These are all needed.

This will force the use of a cache file.  Since we don't know the # threads, I added
a file open counter and flush the cache file when that goes down to zero.

Note:  Previously, I had noticed that one thread may have a delayed open/write/close
which made using a file open counter problematic as all others may close before this
thread opened.  That does not appear to be happening now - at least under this
scenario.
trel pushed a commit that referenced this issue Nov 12, 2021
@JustinKyleJames @trel
[#1980] Fix server upload after server redirect.
e6265c7 
For now, on redirect the plugin can't get file size, #threads, requested #threads,
or oprType.  These are all needed.

This will force the use of a cache file.  Since we don't know the # threads, I added
a file open counter and flush the cache file when that goes down to zero.

Note:  Previously, I had noticed that one thread may have a delayed open/write/close
which made using a file open counter problematic as all others may close before this
thread opened.  That does not appear to be happening now - at least under this
scenario.
@JustinKyleJames
Copy link
Contributor Author

The temporary work-around has been implemented in both 4-2-stable and master. Since it is now working but the permanent solution has not been implemented, I am changing the tag from bug to enhancement.

@JonathanMELIUS JonathanMELIUS mentioned this issue Apr 20, 2023
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JustinKyleJames JustinKyleJames

Labels
consortium-member enhancement
Milestone
No milestone
Development

No branches or pull requests
2 participants
@trel @JustinKyleJames