Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: how do I change the max file upload size in ParseMultipartForm? #23165

Closed
StevenZack opened this issue Dec 18, 2017 · 7 comments

Comments

Projects
None yet
6 participants
@StevenZack
Copy link

commented Dec 18, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

Go version : 1.9.2
GoMobile version : b28c5379a52d34717fd23307abe34f933d42850a

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Dev on Windows 10 x86_64
Run on Android 7.0 arm64

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

I wanna start a web server on my Android Phone, so I wrote a go package , here's the code:

package me

import (
	"fmt"
	"io"
	"net/http"
	"os"
)

var (
	Path string
)

func Start(pkg string) {
	go run(pkg)
}
func run(pkg string) {
	Path = pkg
	http.HandleFunc("/", home)
	http.HandleFunc("/upload", upload)
	http.ListenAndServe(":8090", nil)
}
func home(w http.ResponseWriter, r *http.Request) {
	fmt.Fprint(w, `<!DOCTYPE html>
<html>
<head>
	<title></title>
</head>
<body>
<form action="/upload" enctype="multipart/form-data" method="post">
	<input type="file" name="file">
	<input type="submit" name="submit">
</form>
</body>
</html>`)
}
func upload(w http.ResponseWriter, r *http.Request) {
	r.ParseMultipartForm(32 << 20)
	file, handler, err := r.FormFile("file")
	if err != nil {
		fmt.Println(err)
		return
	}
	defer file.Close()
	fmt.Fprintf(w, "%v", handler.Header)
	f, err := os.OpenFile(Path+handler.Filename, os.O_WRONLY|os.O_CREATE, 0666)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer f.Close()
	io.Copy(f, file)
}

Then I build a me.aar file , using this command:
gomobile.exe bind --target=android/arm,android/arm64 me

Then I import me.aar file into my android project , and start the web server:

public class MainActivity extends AppCompatActivity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
            if (checkSelfPermission(Manifest.permission.WRITE_EXTERNAL_STORAGE)== PackageManager.PERMISSION_GRANTED) {
                Me.start("/sdcard/");
            } else {
                Log.v(TAG,"Permission is revoked");
                ActivityCompat.requestPermissions(MainActivity.this, new String[]{Manifest.permission.CAMERA}, 38);
            }
        }else {
            Me.start("/sdcard/");
        }
    }
}

I gave it internet permission, write-external-storage permission.

Then I start my app, browse http://192.168.0.152:8090 on my PC , I saw the home page!
But when I upload a file that bigger than 32M to my phone , it failed.
(Files that smaller than 32M works great )

What did you expect to see?

Uploading a file that bigger than 32M to my phone succeed.

What did you see instead?

I upload a file that bigger than 32M to my phone , it failed.

@gopherbot gopherbot added this to the Unreleased milestone Dec 18, 2017

@gopherbot gopherbot added the mobile label Dec 18, 2017

@odeke-em odeke-em changed the title x/mobile: net/http: how do I change the max file upload size in ParseMultipartForm? Dec 18, 2017

@odeke-em odeke-em removed the mobile label Dec 18, 2017

@odeke-em

This comment has been minimized.

Copy link
Member

commented Dec 18, 2017

@StevenZack thank you for reporting this and welcome to Go!

So in your code you have this line r.ParseMultipartForm(32 << 20) which limits the file to 32MiB which is what your issue is about
screen shot 2017-12-18 at 12 04 01 am

If you'd like to upload larger files, you'll need to up that limit and also I'd personally advise checking for an error e.g

if err := r.ParseMultipartForm(128 << 20); err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
    return
}

and you can learn more about ParseMultipartForm from the docs at https://golang.org/pkg/net/http/#Request.ParseMultipartForm.
I'll close this issue since my response effectively answers your question of why your code fails to parse more than 32MiB given that that's the limit that you set.

Please feel free to reopen or report more issues if symptoms persist.

@odeke-em odeke-em closed this Dec 18, 2017

@StevenZack

This comment has been minimized.

Copy link
Author

commented Dec 18, 2017

Thanks for your help , but this issue still confuze me.

I tried changing 32<<20 to a larger number (like 10Gib I tried), but it comes to another problem:

While uploading a 3GiB file to my phone ( since it's smaller than the maxMem value 10Gib , it just directly store tmp file in my phone memery )
What I saw is : my phone memery usage was ceaselessly increasing , when it reach to the max memery size of my phone( 2GiB ), my App was killed by Android kernel.

So, using that solution do helped me to upload a file larger that 32M , but I still cannot upload a file larger than my phone's valid memery remains ( usually it's 800M )

I think setting maxMem to 32MiB is totally OK . 32MiB maxMem works on linux , windows , should also work on Android .

I changed my code , to find out whats going on :

	if e := r.ParseMultipartForm(32 << 20); e != nil {
		http.Error(w, e.Error(), http.StatusBadRequest)
		fmt.Println("parse err", e.Error())
		return
	}

And I got error like this when I upload a file larger than 32M:
I/GoLog: parse err open /data/local/tmp/multipart-097615123: permission denied

Maybe it's a permission problem , I don't know .
You guys are incredible , I wish you can help me.

@mvdan

This comment has been minimized.

Copy link
Member

commented Dec 18, 2017

@StevenZack we don't use the issue tracker for questions, so you're more likely to find answers here: https://golang.org/wiki/Questions.

@AlexRouSg

This comment has been minimized.

Copy link
Contributor

commented Dec 18, 2017

@mvdan

Android does not allow access to /data/local/tmp unless you root the device (huge security risk to do so) and since I don't see a way of changing http's tmp dir I believe this might be a bug.

@mvdan

This comment has been minimized.

Copy link
Member

commented Dec 18, 2017

If there is a bug in http, please do file a new issue with clear details and a small program to reproduce it.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Dec 18, 2017

net/http uses mime/multipart which uses io/ioutil.TempDir which uses https://golang.org/pkg/os/#TempDir which lets you set $TMPDIR to something else if you want.

@AlexRouSg

This comment has been minimized.

Copy link
Contributor

commented Dec 19, 2017

@StevenZack

os.TempDir uses the environment var $TMPDIR which you can set through os.Setenv https://golang.org/pkg/os/#Setenv

@golang golang locked and limited conversation to collaborators Dec 19, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.