Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize memory usage when stream flush and save #659

Merged
merged 8 commits into from
Oct 5, 2020

Conversation

snowcrumble
Copy link
Contributor

PR Details

Optimize memory usage when stream flush and save

Description

  1. Record streams in File when new a stream
  2. Do not close temp file when flush stream
  3. use io.Copy to copy from stream buffer reader into zip writer.

Related Issue

#650

Motivation and Context

Avoid instantaneous memory usage.

How Has This Been Tested

I fix a OOM problem in k8s.

Types of changes

  • Docs change / refactoring / dependency upgrade
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@xuri xuri added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 7, 2020
file.go Outdated Show resolved Hide resolved
go.sum Outdated Show resolved Hide resolved
Copy link
Member

@xuri xuri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @snowcrumble, thanks for your PR. I've left some comments. Please make sure to pass the go test.

@codecov-commenter
Copy link

codecov-commenter commented Jul 8, 2020

Codecov Report

Merging #659 into v2 will decrease coverage by 0.05%.
The diff coverage is 52.63%.

Impacted file tree graph

@@            Coverage Diff             @@
##               v2     #659      +/-   ##
==========================================
- Coverage   95.59%   95.53%   -0.06%     
==========================================
  Files          31       31              
  Lines        8442     8441       -1     
==========================================
- Hits         8070     8064       -6     
- Misses        225      229       +4     
- Partials      147      148       +1     
Impacted Files Coverage Δ
excelize.go 95.85% <ø> (ø)
file.go 84.09% <35.71%> (-9.16%) ⬇️
stream.go 87.17% <100.00%> (+0.83%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f2b8798...73b6edd. Read the comment docs.

@snowcrumble snowcrumble requested a review from xuri July 8, 2020 02:06
@Paxa
Copy link

Paxa commented Jul 27, 2020

this change will be very helpful for me

Can you give usage example please? (example in code worked)

I found few things not working in streaming API:

  1. file.SetSheetName(file.GetSheetName(1), "TODO") - has no effect, file.NewStreamWriter will work only with Sheet1 and result will have Sheet1. As workaround I added new Sheet and made it active
  2. using time in writer.SetRow(...) will print it as 44040,0976334606, as workaround I set custom style with date formatting
dateFormat := "[$-409]m/d/yy hh:mm;@"
dateStyle, err := file.NewStyle(&excelize.Style{CustomNumFmt: &dateFormat})

for ... {
  excelRow = append(excelRow, excelize.Cell{Value: val, StyleID: dateFormat})
}
  1. file.SetColWidth has no effect (already reported by others)

@xuri xuri merged commit 9316028 into qax-os:v2 Oct 5, 2020
EugeneAndrosovPaser pushed a commit to ceearrashee/excelize that referenced this pull request Nov 14, 2020
* use io.Copy from stream temp file to zip Writer

* fix nil

* log

* build

* delete log

* fix compatibility for office

* Update go module

Co-authored-by: lijingfeng <lijingfeng@laiye.com>
Co-authored-by: xuri <xuri.me@gmail.com>
jenbonzhang pushed a commit to jenbonzhang/excelize that referenced this pull request Oct 22, 2023
* use io.Copy from stream temp file to zip Writer

* fix nil

* log

* build

* delete log

* fix compatibility for office

* Update go module

Co-authored-by: lijingfeng <lijingfeng@laiye.com>
Co-authored-by: xuri <xuri.me@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants