net/http: how do I change the max file upload size in ParseMultipartForm? #23165

StevenZack · Dec 18, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (`go version`)?

Go version : 1.9.2
GoMobile version : b28c5379a52d34717fd23307abe34f933d42850a

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

Dev on Windows 10 x86_64
Run on Android 7.0 arm64

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

I wanna start a web server on my Android Phone, so I wrote a go package , here's the code:

package me

import (
	"fmt"
	"io"
	"net/http"
	"os"
)

var (
	Path string
)

func Start(pkg string) {
	go run(pkg)
}
func run(pkg string) {
	Path = pkg
	http.HandleFunc("/", home)
	http.HandleFunc("/upload", upload)
	http.ListenAndServe(":8090", nil)
}
func home(w http.ResponseWriter, r *http.Request) {
	fmt.Fprint(w, `<!DOCTYPE html>
<html>
<head>
	<title></title>
</head>
<body>
<form action="/upload" enctype="multipart/form-data" method="post">
	<input type="file" name="file">
	<input type="submit" name="submit">
</form>
</body>
</html>`)
}
func upload(w http.ResponseWriter, r *http.Request) {
	r.ParseMultipartForm(32 << 20)
	file, handler, err := r.FormFile("file")
	if err != nil {
		fmt.Println(err)
		return
	}
	defer file.Close()
	fmt.Fprintf(w, "%v", handler.Header)
	f, err := os.OpenFile(Path+handler.Filename, os.O_WRONLY|os.O_CREATE, 0666)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer f.Close()
	io.Copy(f, file)
}

Then I build a me.aar file , using this command:
gomobile.exe bind --target=android/arm,android/arm64 me

Then I import me.aar file into my android project , and start the web server:

public class MainActivity extends AppCompatActivity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
            if (checkSelfPermission(Manifest.permission.WRITE_EXTERNAL_STORAGE)== PackageManager.PERMISSION_GRANTED) {
                Me.start("/sdcard/");
            } else {
                Log.v(TAG,"Permission is revoked");
                ActivityCompat.requestPermissions(MainActivity.this, new String[]{Manifest.permission.CAMERA}, 38);
            }
        }else {
            Me.start("/sdcard/");
        }
    }
}

I gave it internet permission, write-external-storage permission.

Then I start my app, browse http://192.168.0.152:8090 on my PC , I saw the home page!
But when I upload a file that bigger than 32M to my phone , it failed.
(Files that smaller than 32M works great )

What did you expect to see?

Uploading a file that bigger than 32M to my phone succeed.

What did you see instead?

I upload a file that bigger than 32M to my phone , it failed.

odeke-em · Dec 18, 2017

@StevenZack thank you for reporting this and welcome to Go!

So in your code you have this line r.ParseMultipartForm(32 << 20) which limits the file to 32MiB which is what your issue is about

If you'd like to upload larger files, you'll need to up that limit and also I'd personally advise checking for an error e.g

if err := r.ParseMultipartForm(128 << 20); err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
    return
}

and you can learn more about ParseMultipartForm from the docs at https://golang.org/pkg/net/http/#Request.ParseMultipartForm.
I'll close this issue since my response effectively answers your question of why your code fails to parse more than 32MiB given that that's the limit that you set.

Please feel free to reopen or report more issues if symptoms persist.

StevenZack · Dec 18, 2017

Thanks for your help , but this issue still confuze me.

I tried changing 32<<20 to a larger number (like 10Gib I tried), but it comes to another problem:

While uploading a 3GiB file to my phone ( since it's smaller than the maxMem value 10Gib , it just directly store tmp file in my phone memery )
What I saw is : my phone memery usage was ceaselessly increasing , when it reach to the max memery size of my phone( 2GiB ), my App was killed by Android kernel.

So, using that solution do helped me to upload a file larger that 32M , but I still cannot upload a file larger than my phone's valid memery remains ( usually it's 800M )

I think setting maxMem to 32MiB is totally OK . 32MiB maxMem works on linux , windows , should also work on Android .

I changed my code , to find out whats going on :

	if e := r.ParseMultipartForm(32 << 20); e != nil {
		http.Error(w, e.Error(), http.StatusBadRequest)
		fmt.Println("parse err", e.Error())
		return
	}

And I got error like this when I upload a file larger than 32M:
I/GoLog: parse err open /data/local/tmp/multipart-097615123: permission denied

Maybe it's a permission problem , I don't know .
You guys are incredible , I wish you can help me.

mvdan · Dec 18, 2017

@StevenZack we don't use the issue tracker for questions, so you're more likely to find answers here: https://golang.org/wiki/Questions.

AlexRouSg · Dec 18, 2017

@mvdan

Android does not allow access to /data/local/tmp unless you root the device (huge security risk to do so) and since I don't see a way of changing http's tmp dir I believe this might be a bug.

mvdan · Dec 18, 2017

If there is a bug in http, please do file a new issue with clear details and a small program to reproduce it.

bradfitz · Dec 18, 2017

net/http uses mime/multipart which uses io/ioutil.TempDir which uses https://golang.org/pkg/os/#TempDir which lets you set $TMPDIR to something else if you want.

AlexRouSg · Dec 19, 2017

@StevenZack

os.TempDir uses the environment var $TMPDIR which you can set through os.Setenv https://golang.org/pkg/os/#Setenv

gopherbot added this to the Unreleased milestone Dec 18, 2017

gopherbot added the mobile label Dec 18, 2017

odeke-em changed the title ~~x/mobile:~~ net/http: how do I change the max file upload size in ParseMultipartForm? Dec 18, 2017

odeke-em removed the mobile label Dec 18, 2017

odeke-em closed this Dec 18, 2017

golang locked and limited conversation to collaborators Dec 19, 2018

gopherbot added the FrozenDueToAge label Dec 19, 2018

golang/go

net/http: how do I change the max file upload size in ParseMultipartForm? #23165

net/http: how do I change the max file upload size in ParseMultipartForm? #23165

StevenZack commented Dec 18, 2017 •

edited

gopherbot added this to the Unreleased milestone Dec 18, 2017

gopherbot added the mobile label Dec 18, 2017

odeke-em changed the title x/mobile: net/http: how do I change the max file upload size in ParseMultipartForm? Dec 18, 2017

odeke-em removed the mobile label Dec 18, 2017

This comment has been minimized.

odeke-em commented Dec 18, 2017

odeke-em closed this Dec 18, 2017

This comment has been minimized.

StevenZack commented Dec 18, 2017 •

edited

This comment has been minimized.

mvdan commented Dec 18, 2017

This comment has been minimized.

AlexRouSg commented Dec 18, 2017 •

edited

This comment has been minimized.

mvdan commented Dec 18, 2017

This comment has been minimized.

bradfitz commented Dec 18, 2017

This comment has been minimized.

AlexRouSg commented Dec 19, 2017

golang locked and limited conversation to collaborators Dec 19, 2018

gopherbot added the FrozenDueToAge label Dec 19, 2018

golang/go

Track tasks and feature requests

net/http: how do I change the max file upload size in ParseMultipartForm? #23165

Comments

StevenZack commented Dec 18, 2017 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

gopherbot added this to the Unreleased milestone Dec 18, 2017

gopherbot added the mobile label Dec 18, 2017

odeke-em changed the title x/mobile: net/http: how do I change the max file upload size in ParseMultipartForm? Dec 18, 2017

odeke-em removed the mobile label Dec 18, 2017

This comment has been minimized.

odeke-em commented Dec 18, 2017

odeke-em closed this Dec 18, 2017

This comment has been minimized.

StevenZack commented Dec 18, 2017 • edited

I tried changing 32<<20 to a larger number (like 10Gib I tried), but it comes to another problem:

So, using that solution do helped me to upload a file larger that 32M , but I still cannot upload a file larger than my phone's valid memery remains ( usually it's 800M )

I changed my code , to find out whats going on :

This comment has been minimized.

mvdan commented Dec 18, 2017

This comment has been minimized.

AlexRouSg commented Dec 18, 2017 • edited

This comment has been minimized.

mvdan commented Dec 18, 2017

This comment has been minimized.

bradfitz commented Dec 18, 2017

This comment has been minimized.

AlexRouSg commented Dec 19, 2017

golang locked and limited conversation to collaborators Dec 19, 2018

gopherbot added the FrozenDueToAge label Dec 19, 2018

StevenZack commented Dec 18, 2017 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

StevenZack commented Dec 18, 2017 •

edited

AlexRouSg commented Dec 18, 2017 •

edited