fix bug #1085 #1122

johnwonder · 2020-10-23T13:39:52Z

我测试上传了6个多G的文件没问题 #1085

JohnNiang · 2020-10-23T14:56:38Z

#1085 中出现的 OOM 错误主要是因为我在获取 run.halo.app.security.filter.AbstractAuthenticationFilter.getTokenFromRequest(AbstractAuthenticationFilter.java:250) 的时候，jetty 解析参数的时候将上传的文件流输出在了 byte[] 中，导致 Requested array size exceeds VM limit。

理论上这是 jetty 的问题，不过通过这样的方式修复我们还需要调试一下。

src/main/resources/application-dev.yaml

JohnNiang · 2020-10-23T16:44:15Z

测试通过。

JohnNiang · 2020-10-23T16:53:24Z

我刚刚测试 10G 文件上传，没有出现问题。

2020-10-24 00:51:59.381 DEBUG 156870 --- [tp1993153814-60] run.halo.app.core.ControllerLogAop       : Request URL: [http://localhost:8090/api/admin/attachments/upload], URI: [/api/admin/attachments/upload], Request Method: [POST], IP: [0:0:0:0:0:0:0:1]
2020-10-24 00:51:59.381 DEBUG 156870 --- [tp1993153814-60] r.h.a.s.impl.AttachmentServiceImpl       : Starting uploading... type: [LOCAL], file: [bigfile]
2020-10-24 00:51:59.382 DEBUG 156870 --- [tp1993153814-60] r.h.app.handler.file.LocalFileHandler    : Base name: [bigfile-328bccaf764b4161b89ba7bf4a096437], extension: [] of original filename: [bigfile]
2020-10-24 00:51:59.382  INFO 156870 --- [tp1993153814-60] r.h.app.handler.file.LocalFileHandler    : Uploading file: [bigfile]to directory: [/home/johnniang/halo-dev/upload/2020/10/bigfile-328bccaf764b4161b89ba7bf4a096437.]
2020-10-24 00:52:09.710  INFO 156870 --- [tp1993153814-60] r.h.app.handler.file.LocalFileHandler    : Uploaded file: [bigfile] to directory: [/home/johnniang/halo-dev/upload/2020/10/bigfile-328bccaf764b4161b89ba7bf4a096437.] successfully
2020-10-24 00:52:09.717 DEBUG 156870 --- [tp1993153814-60] r.h.a.s.impl.AttachmentServiceImpl       : Attachment type: [LOCAL]
2020-10-24 00:52:09.717 DEBUG 156870 --- [tp1993153814-60] r.h.a.s.impl.AttachmentServiceImpl       : Upload result: [UploadResult(filename=bigfile, filePath=upload/2020/10/bigfile-328bccaf764b4161b89ba7bf4a096437., key=upload/2020/10/bigfile-328bccaf764b4161b89ba7bf4a096437., thumbPath=upload/2020/10/bigfile-328bccaf764b4161b89ba7bf4a096437., suffix=, mediaType=application/octet-stream, size=10737418240, width=null, height=null)]
2020-10-24 00:52:09.717 DEBUG 156870 --- [tp1993153814-60] r.h.a.s.impl.AttachmentServiceImpl       : Creating attachment: [Attachment(id=null, name=bigfile, path=upload/2020/10/bigfile-328bccaf764b4161b89ba7bf4a096437., fileKey=upload/2020/10/bigfile-328bccaf764b4161b89ba7bf4a096437., thumbPath=upload/2020/10/bigfile-328bccaf764b4161b89ba7bf4a096437., mediaType=application/octet-stream, suffix=, width=null, height=null, size=10737418240, type=LOCAL)]
Hibernate: select count(attachment0_.id) as col_0_0_ from attachments attachment0_ where attachment0_.path=?
Hibernate: insert into attachments (id, create_time, update_time, file_key, height, media_type, name, path, size, suffix, thumb_path, type, width) values (null, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
2020-10-24 00:52:09.720 DEBUG 156870 --- [tp1993153814-60] run.halo.app.core.ControllerLogAop       : AttachmentController.uploadAttachment Response: [{"id":10,"name":"bigfile","path":"http://localhost:8090/upload/2020/10/bigfile-328bccaf764b4161b89ba7bf4a096437.","fileKey":"upload/2020/10/bigfile-328bccaf764b4161b89ba7bf4a096437.","thumbPath":"http://localhost:8090/upload/2020/10/bigfile-328bccaf764b4161b89ba7bf4a096437.","mediaType":"application/octet-stream","suffix":"","width":0,"height":0,"size":10737418240,"type":"LOCAL","createTime":1603471929718}], usage: [10339]ms
2020-10-24 00:52:10.358 DEBUG 156870 --- [tp1993153814-60] run.halo.app.filter.LogFilter            : Ending   url: [http://localhost:8090/api/admin/attachments/upload], method: [POST], ip: [0:0:0:0:0:0:0:1], status: [200], usage: [57047] ms

JohnNiang · 2020-10-23T16:54:40Z

@guqing 如果你使用的是非 application-dev.yml 配置文件，需要配置一下：

spring:
  autoconfigure:
    exclude: org.springframework.boot.autoconfigure.web.servlet.MultipartAutoConfiguratio

guqing · 2020-10-23T16:57:15Z

@guqing 如果你使用的是非 application-dev.yml 配置文件，需要配置一下：
spring:
  autoconfigure:
    exclude: org.springframework.boot.autoconfigure.web.servlet.MultipartAutoConfiguratio

sorry忘记看了我刚刚发完日志才看了一下配置文件重新试了一下 8g没问题

ruibaby

建议在 Application 类里面用注解 exclude。不想在配置文件有太多东西。

guqing · 2020-10-23T17:09:50Z

16g也测试通过

JohnNiang · 2020-10-24T00:56:12Z

@guqing 不应该啊，我记得设置的最大上传为 10G 啊😂

JohnNiang · 2020-10-24T01:18:57Z

建议测试一下以下配置是否还起作用：

spring.servlet.multipart.max-file-size=128KB
spring.servlet.multipart.max-request-size=128KB

johnwonder · 2020-10-24T03:31:26Z

@JohnNiang 因为在代码里手动定义了MultipartBean 把autoconfiguration排除了所以在配置文件里定义的配置是失效的可以在手动定义Bean的时候设置maxSize 要用外部配置的话可以定义个confoguration class导入外部配置

guqing · 2020-10-24T05:38:09Z

@JohnNiang 因为在代码里手动定义了MultipartBean 把autoconfiguration排除了所以在配置文件里定义的配置是失效的可以在手动定义Bean的时候设置maxSize 要用外部配置的话可以定义个confoguration class导入外部配置

我觉得可以使用MultiPartProperties

johnwonder · 2020-10-24T05:40:48Z

@JohnNiang 因为在代码里手动定义了MultipartBean 把autoconfiguration排除了所以在配置文件里定义的配置是失效的可以在手动定义Bean的时候设置maxSize 要用外部配置的话可以定义个confoguration class导入外部配置

我觉得可以使用MultiPartProperties

加上了

guqing · 2020-10-24T05:42:16Z

@JohnNiang 因为在代码里手动定义了MultipartBean 把autoconfiguration排除了所以在配置文件里定义的配置是失效的可以在手动定义Bean的时候设置maxSize 要用外部配置的话可以定义个confoguration class导入外部配置

我觉得可以使用MultiPartProperties

加上了
ok看到了

JohnNiang · 2020-10-24T15:21:49Z

src/main/java/run/halo/app/config/WebMvcAutoConfiguration.java

+        MultipartConfigElement multipartConfigElement = multipartProperties.createMultipartConfig();
+        CommonsMultipartResolver resolver = new CommonsMultipartResolver();
+        resolver.setDefaultEncoding("UTF-8");
+        resolver.setMaxUploadSize(multipartConfigElement.getMaxFileSize());


这里的配置似乎设置反了。

resolver.setMaxUploadSize(multipartConfigElement.getMaxRequestSize()); resolver.setMaxUploadSizePerFile(multipartConfigElement.getMaxFileSize())

也建议对 org.springframework.web.multipart.MaxUploadSizeExceededException 异常进行处理。

好的改了

我本来觉得不要catch 这个异常的，因为已经有了multipart配置，可以手动使用
boolean isMultiPart = MimeTypes.Type.MULTIPART_FORM_DATA.is(HttpFields.valueParameters(request.getContentType(), null)); if(isMultiPart && request.getContentLength() > multipartProperties.getMaxFileSize().toBytes())
来判断上传大小，但是想想还是觉得交由框架层面去判断吧

src/main/java/run/halo/app/config/WebMvcAutoConfiguration.java

JohnNiang · 2020-10-25T03:03:11Z

src/main/java/run/halo/app/security/filter/AdminAuthenticationFilter.java

+        try{
+            filterChain.doFilter(request, response);
+        }
+        catch (NestedServletException nestedServletEx){


不推荐在 AdminAuthenticationFilter 中处理这样的异常。

建议在 run.halo.app.core.ControllerExceptionHandler 中添加全局异常处理。

@ExceptionHandler(MaxUploadSizeExceededException.class) @ResponseStatus(HttpStatus.BAD_REQUEST) public BaseResponse<?> handleUploadSizeExceededException(MaxUploadSizeExceededException e) { BaseResponse<Object> response = handleBaseException(e); response.setStatus(HttpStatus.BAD_REQUEST.value()); response.setMessage("当前请求超出最大限制：" + e.getMaxUploadSize() + " bytes"); return response; }

这里有一个前提是：设置 CommonsMultipartResolver 的 resolveLazily 属性为 true，否则无法正常捕捉到异常。

CommonsMultipartResolver resolver = new CommonsMultipartResolver(); resolver.setDefaultEncoding("UTF-8"); resolver.setMaxUploadSize(multipartConfigElement.getMaxRequestSize()); resolver.setMaxUploadSizePerFile(multipartConfigElement.getMaxFileSize()); + resolver.setResolveLazily(true); return resolver;

原来ResolveLazily是这个意思我靠学到了🙊 妙啊

ok,忘了有全局异常处理。。。,感谢指教

JohnNiang · 2020-10-25T05:44:36Z

需要去除未引用的 imports。

JohnNiang · 2020-10-25T05:54:58Z

必要手动设置response.status了吧会被注解处理

这里的 response 是返回给前端判断的，算是业务 status。

guqing · 2020-10-25T05:56:21Z

必要手动设置response.status了吧会被注解处理

这里的 response 是返回给前端判断的，算是业务 status。

我看错了看到response以为是servlet的回复完才看到就删了😂

* fix bug halo-dev#1085 * move exclude to Application class * enable multipart configuration * fixt multipart config error and catch MaxUploadSizeExceededException * fix checkstyle error * fix if not followed by whitespace * fix checkstyle error * modify multipart exception handling to global exception handling * remove unnecessary catch * remove unnecessary catch

XM2510136957 · 2022-05-16T11:09:18Z

Caused by: java.lang.OutOfMemoryError: Java heap space
at java.base/jdk.internal.misc.Unsafe.allocateUninitializedArray(Unsafe.java:1375) ~[na:na]
at java.base/java.lang.StringConcatHelper.newArray(StringConcatHelper.java:494) ~[na:na]
at java.base/java.lang.String.join(String.java:3264) ~[na:na]
at java.base/java.lang.System$2.join(System.java:2438) ~[na:na]
at java.base/java.util.StringJoiner.toString(StringJoiner.java:174) ~[na:na]
at java.base/java.util.stream.Collectors$$Lambda$46/0x800000054.apply(Unknown Source) ~[na:na]
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:686) ~[na:na]
at run.halo.app.utils.FileUtils.readString(FileUtils.java:575) ~[classes!/:1.5.3]
at run.halo.app.service.impl.BackupServiceImpl.importMarkdown(BackupServiceImpl.java:235) ~[classes!/:1.5.3]
at run.halo.app.controller.admin.api.BackupController.backupMarkdowns(BackupController.java:148) ~[classes!/:1.5.3]

JohnNiang · 2022-05-16T14:25:09Z

Caused by: java.lang.OutOfMemoryError: Java heap space at java.base/jdk.internal.misc.Unsafe.allocateUninitializedArray(Unsafe.java:1375) ~[na:na] at java.base/java.lang.StringConcatHelper.newArray(StringConcatHelper.java:494) ~[na:na] at java.base/java.lang.String.join(String.java:3264) ~[na:na] at java.base/java.lang.System$2.join(System.java:2438) ~[na:na] at java.base/java.util.StringJoiner.toString(StringJoiner.java:174) ~[na:na] at java.base/java.util.stream.Collectors$$Lambda$46/0x800000054.apply(Unknown Source) ~[na:na] at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:686) ~[na:na] at run.halo.app.utils.FileUtils.readString(FileUtils.java:575) ~[classes!/:1.5.3] at run.halo.app.service.impl.BackupServiceImpl.importMarkdown(BackupServiceImpl.java:235) ~[classes!/:1.5.3] at run.halo.app.controller.admin.api.BackupController.backupMarkdowns(BackupController.java:148) ~[classes!/:1.5.3]

See #2088

fix bug halo-dev#1085

522f3d5

ruibaby requested review from JohnNiang, ruibaby, LIlGG and guqing October 23, 2020 13:42

ruibaby added the kind/feature Categorizes issue or PR as related to a new feature. label Oct 23, 2020

JohnNiang reviewed Oct 23, 2020

View reviewed changes

src/main/resources/application-dev.yaml Outdated Show resolved Hide resolved

ruibaby requested changes Oct 23, 2020

View reviewed changes

johnwonder requested a review from ruibaby October 23, 2020 23:13

move exclude to Application class

6651276

ruibaby approved these changes Oct 23, 2020

View reviewed changes

enable multipart configuration

9656085

JohnNiang self-requested a review October 24, 2020 08:00

JohnNiang requested changes Oct 24, 2020

View reviewed changes

fixt multipart config error and catch MaxUploadSizeExceededException

8aa4d15

johnwonder requested a review from JohnNiang October 25, 2020 02:16

johnwonder added 2 commits October 25, 2020 10:20

fix checkstyle error

7a509bc

fix if not followed by whitespace

6f34ab0

fix checkstyle error

3b34e59

guqing approved these changes Oct 25, 2020

View reviewed changes

src/main/java/run/halo/app/config/WebMvcAutoConfiguration.java Outdated Show resolved Hide resolved

JohnNiang requested changes Oct 25, 2020

View reviewed changes

modify multipart exception handling to global exception handling

8a0cc1a

johnwonder requested a review from JohnNiang October 25, 2020 05:32

remove unnecessary catch

657fccd

remove unnecessary catch

814ba76

JohnNiang merged commit b5f7425 into halo-dev:master Oct 25, 2020

JohnNiang linked an issue Oct 25, 2020 that may be closed by this pull request

上传文件出现 OOM 错误 #1085

Closed

ruibaby mentioned this pull request Dec 12, 2020

上传文件出现OOM错误 #1195

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix bug #1085 #1122

fix bug #1085 #1122

johnwonder commented Oct 23, 2020

JohnNiang commented Oct 23, 2020

JohnNiang commented Oct 23, 2020

JohnNiang commented Oct 23, 2020

JohnNiang commented Oct 23, 2020

guqing commented Oct 23, 2020

ruibaby left a comment

guqing commented Oct 23, 2020

JohnNiang commented Oct 24, 2020

JohnNiang commented Oct 24, 2020 •

edited

johnwonder commented Oct 24, 2020 •

edited

guqing commented Oct 24, 2020

johnwonder commented Oct 24, 2020

guqing commented Oct 24, 2020

JohnNiang Oct 24, 2020

JohnNiang Oct 24, 2020

johnwonder Oct 25, 2020

johnwonder Oct 25, 2020

JohnNiang Oct 25, 2020 •

edited

guqing Oct 25, 2020 •

edited

johnwonder Oct 25, 2020 •

edited

JohnNiang commented Oct 25, 2020

JohnNiang commented Oct 25, 2020

guqing commented Oct 25, 2020

XM2510136957 commented May 16, 2022

JohnNiang commented May 16, 2022

fix bug #1085 #1122

fix bug #1085 #1122

Conversation

johnwonder commented Oct 23, 2020

JohnNiang commented Oct 23, 2020

JohnNiang commented Oct 23, 2020

JohnNiang commented Oct 23, 2020

JohnNiang commented Oct 23, 2020

guqing commented Oct 23, 2020

ruibaby left a comment

Choose a reason for hiding this comment

guqing commented Oct 23, 2020

JohnNiang commented Oct 24, 2020

JohnNiang commented Oct 24, 2020 • edited

johnwonder commented Oct 24, 2020 • edited

guqing commented Oct 24, 2020

johnwonder commented Oct 24, 2020

guqing commented Oct 24, 2020

JohnNiang Oct 24, 2020

Choose a reason for hiding this comment

JohnNiang Oct 24, 2020

Choose a reason for hiding this comment

johnwonder Oct 25, 2020

Choose a reason for hiding this comment

johnwonder Oct 25, 2020

Choose a reason for hiding this comment

JohnNiang Oct 25, 2020 • edited

Choose a reason for hiding this comment

guqing Oct 25, 2020 • edited

Choose a reason for hiding this comment

johnwonder Oct 25, 2020 • edited

Choose a reason for hiding this comment

JohnNiang commented Oct 25, 2020

JohnNiang commented Oct 25, 2020

guqing commented Oct 25, 2020

XM2510136957 commented May 16, 2022

JohnNiang commented May 16, 2022

JohnNiang commented Oct 24, 2020 •

edited

johnwonder commented Oct 24, 2020 •

edited

JohnNiang Oct 25, 2020 •

edited

guqing Oct 25, 2020 •

edited

johnwonder Oct 25, 2020 •

edited