Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Importing huge data is very CPU and memory intensive #8

Open
tinesoft opened this issue Jan 29, 2020 · 0 comments
Open

[Bug] Importing huge data is very CPU and memory intensive #8

tinesoft opened this issue Jan 29, 2020 · 0 comments

Comments

@tinesoft
Copy link
Owner

Bug Report or Feature Request (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request

Loading huge data file (200K+ entries) is very memory and CPU intensive (almost 100% of cpu used). Make sure that the streaming of each line/row being read is effective.

Tested on a Docker env, capped to 300M of memory and 1 CPU

Spring Versions?

- `spring boot version`: all
- or `spring`, 'spring-test`, and `spring-data-elasticsearch` versions: all

Elasticsearch server Version?

6.7.0

OS Version?

Linux (CentOS)

Repro steps

Import a huge dataset (200K+ entries), on a docker env, capped to 300M and 1 CPU for example.

The log given by the failure

Desired functionality

Mention any other details that might be useful

@tinesoft tinesoft changed the title [Bug] Importing huge is very CPU and memory intensive [Bug] Importing huge data is very CPU and memory intensive Jan 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant