Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Robust large file transfers #67

Closed
12 of 13 tasks
sedwards2009 opened this issue Oct 13, 2017 · 1 comment
Closed
12 of 13 tasks

Robust large file transfers #67

sedwards2009 opened this issue Oct 13, 2017 · 1 comment
Assignees
@sedwards2009
Labels
enhancement prio:high

Comments

@sedwards2009
Copy link
Owner

sedwards2009 commented Oct 13, 2017 •
edited
Loading

Extraterm already has some built in support for transferring files using its show and from commands but the current implementation is closer to a PoC than a robust system for moving data around. It is just enough to move bits of text around. What is needed is a full implementation which supports large files, is robust w.r.t. possible interference (think background jobs writing to the terminal), and is also binary safe.

  • Spooling large files to disk
    • Basic support
    • Flow control
    • Encrypting data on disk. This is needed to prevent Extraterm being a litter-bug with potentially sensitive user data.
    • Deleting data from disk when it is no longer needed.
    • Cleaning up previous data on disk. Needed in the case that Extraterm exits in an ungraceful way. Moved to Detect any old temporary files and clean them up automatically #74
  • Exposing large files via a pseudo web protocol to support streaming to the DOM
  • Robust detection and handling of interference and unexpected crashes and exits at the remote end
    • Downloads via show command
    • Uploads, via from command
  • Downloaded file viewer
  • Streaming support in the viewers
    • Image viewer
@sedwards2009 sedwards2009 self-assigned this Oct 13, 2017
@sedwards2009 sedwards2009 mentioned this issue Oct 13, 2017
Open
sedwards2009 added a commit that referenced this issue Oct 13, 2017
@sedwards2009
Support for spooling large files to disk
b5ff3c4 
issue #67
sedwards2009 added a commit that referenced this issue Oct 13, 2017
@sedwards2009
Fix the serving of files over the pseudo web protocol
10811f9 
issue #67
sedwards2009 added a commit that referenced this issue Oct 17, 2017
@sedwards2009
Fixes for the ReaderWriterFile
b614f36 
issue #67
sedwards2009 added a commit that referenced this issue Oct 19, 2017
@sedwards2009
Better handling of empty reads in the WriterReaderFile
ca6ec77 
issue #67
sedwards2009 added a commit that referenced this issue Oct 25, 2017
@sedwards2009
Add flow control when downloading files
a74579b 
issue #67
sedwards2009 added a commit that referenced this issue Oct 26, 2017
@sedwards2009
Store the bulk downloaded files in the user's tmp directory and clean…
26852c1 
… it up when done.

issue #67
sedwards2009 added a commit that referenced this issue Oct 27, 2017
@sedwards2009
Refactor the imports in the TextViewer
dc4ca21 
issue #67
sedwards2009 added a commit that referenced this issue Oct 27, 2017
@sedwards2009
Convert the TextViewer to understand a BulkFileHandle
fee0188 
issue #67
sedwards2009 added a commit that referenced this issue Oct 28, 2017
@sedwards2009
Remove the unneeded BulkFileWebProtocol file
fcc9c20 
issue #67
sedwards2009 added a commit that referenced this issue Nov 2, 2017
@sedwards2009
Remove the custom protocol. It is too limited in practice and require…
4785ad3 
…s a HTTP server anyway.

issue #67
sedwards2009 added a commit that referenced this issue Nov 2, 2017
@sedwards2009
Pass around Bulk File handles when fetching frames. Make from comma…
65a9058 
…nd binary safe.

issue #67
sedwards2009 added a commit that referenced this issue Nov 2, 2017
@sedwards2009
Move the BulkFileUploader out into its own file
24374bd 
issue #67
sedwards2009 added a commit that referenced this issue Nov 2, 2017
@sedwards2009
Clean up a Bulk File Uploader a bit
5bba7d6 
issue #67
sedwards2009 added a commit that referenced this issue Nov 4, 2017
@sedwards2009
Improve the error handling in the from command
59e8174 
issue #67
sedwards2009 added a commit that referenced this issue Nov 6, 2017
@sedwards2009
Repair the from command in combination with terminal output.
abc4f0a 
issue #67
sedwards2009 added a commit that referenced this issue Nov 6, 2017
@sedwards2009
Fix a small bug in the from when dealing with metadata
6597b13 
issue #67
sedwards2009 added a commit that referenced this issue Nov 6, 2017
@sedwards2009
Add missing declaration file for get-uri.
2d367b6 
issue #67
sedwards2009 added a commit that referenced this issue Nov 8, 2017
@sedwards2009
Enforce a minimum size on the size of data chunks we send to storage …
1648e50 
…across IPC.

issue #67
sedwards2009 added a commit that referenced this issue Nov 9, 2017
@sedwards2009
Remove debug code and set reasonable buffer sizes for bulk files
4d021e2 
issue #67
sedwards2009 added a commit that referenced this issue Nov 12, 2017
@sedwards2009
Add utility for detecting objects which are disposable.
458604f 
issue #67
sedwards2009 added a commit that referenced this issue Nov 12, 2017
@sedwards2009
Docs for BulkFileHandle
f7c50c6 
issue #67
sedwards2009 added a commit that referenced this issue Nov 12, 2017
@sedwards2009
Remove temp bulk files from disk when they are no longer needed
59f7cc5 
issue #67
sedwards2009 added a commit that referenced this issue Nov 13, 2017
@sedwards2009
Encrypt the bulk files on disk
26ded79 
issue #67
sedwards2009 added a commit that referenced this issue Nov 14, 2017
@sedwards2009
Upgrade Vue to 2.0.6
d7dfd6d 
issue #67
sedwards2009 added a commit that referenced this issue Nov 15, 2017
@sedwards2009
Upgrade to Vue 2.5.3 (for real)
a8290f1 
issue #67
sedwards2009 added a commit that referenced this issue Nov 18, 2017
@sedwards2009
Introduce a decorator to help make Custom Elements. Use it on ScrollBar
8c21e70 
issue #67
sedwards2009 added a commit that referenced this issue Dec 30, 2017
@sedwards2009
Delay the display of the ETA in the download viewer
754bd9c 
issue #67
sedwards2009 added a commit that referenced this issue Dec 30, 2017
@sedwards2009
Fix the vertical alignment of the text in the Download viewer
433a488 
issue #67
sedwards2009 added a commit that referenced this issue Dec 30, 2017
@sedwards2009
Turn off local echo on the TTY when performing a download
ba4880a 
If local echo isn't turned off, then any accidental key presses from the user will break the download.

issue #67
sedwards2009 added a commit that referenced this issue Dec 30, 2017
@sedwards2009
Improve the tooltip on the Download viewer
15fe33a 
issue #67
sedwards2009 added a commit that referenced this issue Jan 1, 2018
@sedwards2009
Extract the Download status Vue component into its own element
cb1f176 
issue #67
sedwards2009 added a commit that referenced this issue Jan 2, 2018
@sedwards2009
Rename the download-viewer.scss to the new gui-file-transfer-progress…
a0e8c60 
….scss

issue #67
sedwards2009 added a commit that referenced this issue Jan 2, 2018
@sedwards2009
Add a simple progress bar when uploading via the from command.
98063db 
issue #67
sedwards2009 added a commit that referenced this issue Jan 6, 2018
@sedwards2009
Convert the PTY stuff to use the newer event emitter system
116432e 
issue #67
sedwards2009 added a commit that referenced this issue Jan 12, 2018
@sedwards2009
PtyIpcBridge doesn't need to know what a EtTerminal is.
abbd4d2 
issue #67
sedwards2009 added a commit that referenced this issue Jan 15, 2018
@sedwards2009
Refactor the BulkFileUploader in preparation of the coming changes
11c4d5a 
issue #67
sedwards2009 added a commit that referenced this issue Jan 16, 2018
@sedwards2009
Add flow control when uploading files on Linux/OSX.
a797e4c 
issue #67
sedwards2009 added a commit that referenced this issue Jan 19, 2018
@sedwards2009
Refactor ptyserver2.py a bit and do async writes to PTYs
7449809 
issue #67
sedwards2009 added a commit that referenced this issue Jan 19, 2018
@sedwards2009
Add flow control when writing to the a PTY via the proxy PTY connector.
b4307fb 
issue #67
sedwards2009 added a commit that referenced this issue Jan 19, 2018
@sedwards2009
Filter the keyboard input to the PTY when an upload is running.
76bdbd0 
issue #67
sedwards2009 added a commit that referenced this issue Jan 19, 2018
@sedwards2009
Show the correct info in the upload progress bar.
3d007e0 
issue #67
sedwards2009 added a commit that referenced this issue Jan 20, 2018
@sedwards2009
Update the protocol used to send data to the from command
6b9af19 
issue #67
sedwards2009 added a commit that referenced this issue Jan 21, 2018
@sedwards2009
Implement new protocol for from with extra error checking
2f44eab 
issue #67
sedwards2009 added a commit that referenced this issue Jan 23, 2018
@sedwards2009
Increase the size of the chunks sent to the from command
8f53a5e 
issue #67
sedwards2009 added a commit that referenced this issue Jan 24, 2018
@sedwards2009
Fix up the flow control on the sending side for uploads/from
6d11760 
issue #67
sedwards2009 added a commit that referenced this issue Jan 25, 2018
@sedwards2009
Send each line a one chunk
681d58d 
issue #67
sedwards2009 added a commit that referenced this issue Jan 27, 2018
@sedwards2009
Improve debug for BulkFileUploader.ts
a2f4345 
issue #67
sedwards2009 added a commit that referenced this issue Jan 27, 2018
@sedwards2009
Rewrite BulkFileUploader and support aborting.
be7bb9a 
issue #67
sedwards2009 added a commit that referenced this issue Jan 28, 2018
@sedwards2009
Correct the data size calculation in the PTY proxy code.
4bd2f88 
issue #67
sedwards2009 added a commit that referenced this issue Jan 28, 2018
@sedwards2009
Throttle the uploading of files from the main to render process, and …
f20f140 
…prevent memory use from exploding.

issue #67
sedwards2009 added a commit that referenced this issue Jan 30, 2018
@sedwards2009
Stopo reading the file when an upload is aborted.
c96a55a 
issue #67
sedwards2009 added a commit that referenced this issue Feb 1, 2018
@sedwards2009
Improve the edges of the upload progress dialog
50bc6c6 
issue #67
sedwards2009 added a commit that referenced this issue Feb 4, 2018
@sedwards2009
Make from more compatible with cygwin and systems with dodgy tty la…
7f33df8 
…yers.

issue #67
@sedwards2009 sedwards2009 mentioned this issue Feb 10, 2018
Open
@sedwards2009
Copy link
Owner Author

This will be in 0.30.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sedwards2009 sedwards2009

Labels
enhancement prio:high
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sedwards2009