-
Notifications
You must be signed in to change notification settings - Fork 327
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
q.site.upload
is slow
#982
Comments
@srini-x I can't seem to assign you, please just assign yourself. |
Looks like the httpx async client is slower than the regular client: package main
import (
"io"
"log"
"net/http"
"os"
)
func upload(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(32 << 20); err != nil { // 32 MB
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
form := r.MultipartForm
files, ok := form.File["files"]
if !ok {
http.Error(w, "no files", http.StatusBadRequest)
return
}
for _, file := range files {
src, err := file.Open()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
defer src.Close()
dst, err := os.OpenFile(file.Filename, os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
defer dst.Close()
io.Copy(dst, src)
}
}
func main() {
http.HandleFunc("/", upload)
log.Fatal(http.ListenAndServe(":8080", nil)) import httpx
import time
import os
files = ['data.dump']
print('uploading...')
start = time.time()
res = httpx.post('http://localhost:8080/', files=[('files', (os.path.basename(f), open(f, 'rb'))) for f in files])
print(f'upload time: {time.time() - start}, status: {res.status_code} {res.text}')
import asyncio
import httpx
import time
import os
files = ['data.dump']
async def main():
async with httpx.AsyncClient() as client:
print('uploading...')
start = time.time()
res = await client.post('http://localhost:8080/',
files=[('files', (os.path.basename(f), open(f, 'rb'))) for f in files])
print(f'upload time: {time.time() - start}, status: {res.status_code} {res.text}')
asyncio.run(main())
|
Maybe related: encode/httpx#838 |
Looks like they are not eager to improve speed soon: |
FYI: workaround / alternate feature that might help circumventing this issue: https://github.com/h2oai/wave/blob/master/website/docs/files.md#serving-files-directly-from-the-wave-server Will be out in the next release. |
@lo5 looks good, will this also work in h2o-cloud? |
Closed: |
We could move the files within the FS in if wave app and wave server are located on the same machine (which is very common anyway) and dodge pushing it through HTTP. Wdyt @lo5? |
…rver are running on the same machine. #982
…rver are running on the same machine. #982
Wave SDK Version, OS
0.17.0, Linux
Actual behavior
We are dealing with some large files (predictions, model weights etc.) with a couple hundred MB, which we want to make available for download.
I'm running Wave locally, and with the following code:
I get
i.e. it takes ~ 33 seconds for a 100MB file to start downloading.
Expected behavior
Since the file only has to be copied to a location on the same machine, I'd expect the download to start almost immediately.
Steps To Reproduce
Start
waved
locally.head -c 100MB < /dev/urandom > data.dump
Write
app.py
:wave run app
The text was updated successfully, but these errors were encountered: