Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When creating a nonuniformly distributed dataset the source goes out of memory #119

Closed
ale93p opened this issue Aug 7, 2019 · 0 comments
Closed

When creating a nonuniformly distributed dataset the source goes out of memory #119

ale93p opened this issue Aug 7, 2019 · 0 comments
Labels
Conclusion: Solved The issue has been solved Platform: All The issue is related to all the applications Priority: Critical The issue is a blocker Type: Bug Something isn't working

Comments

@ale93p
Copy link
Owner

ale93p commented Aug 7, 2019 •
edited

Issue Description

When genereting a large enough dataset, e.g. 100 values, in a nonuniform distribution the source goes out of memory.
This because it will create an array of 100!elements.

Proposed Solution (if you have any)

Use random gaussian index generation instead of creating a static array.
@ale93p ale93p added Type: Bug Something isn't working Priority: Critical The issue is a blocker Platform: All The issue is related to all the applications labels Aug 7, 2019
ale93p added a commit that referenced this issue Aug 22, 2019
@ale93p
Integrated index generation in DataGenerator; fix #119
d295a4a
@ale93p ale93p mentioned this issue Aug 22, 2019
Merged
4 tasks
ale93p added a commit that referenced this issue Aug 22, 2019
@ale93p
Improved Data Generator (#126)
dbcc4b9 
* Integrated index generation in DataGenerator; fix #119

* [Storm] Implemented new DataGenerator

* [v0.6.1-dev] bump development version

* [Flink] Implemented new data generator

* [Storm] Implemented new data generator

* [Flink] Implemented new data generator

* [All] Fixed conversion from bytes to string

* [All] Cleaned code
@ale93p ale93p added the Conclusion: Solved The issue has been solved label Aug 22, 2019
@ale93p ale93p closed this as completed Aug 22, 2019
@ale93p ale93p reopened this Aug 23, 2019
@ale93p ale93p closed this as completed in d8f5056 Sep 10, 2019
ale93p added a commit that referenced this issue Dec 11, 2019
@ale93p
[v0.7.0] Redisgn Data Generator and External Kafka Source Support (#131)
b6eebb8 
* Improved Data Generator (#126)

* Integrated index generation in DataGenerator; fix #119

* [Storm] Implemented new DataGenerator

* [v0.6.1-dev] bump development version

* [Flink] Implemented new data generator

* [Storm] Implemented new data generator

* [Flink] Implemented new data generator

* [All] Fixed conversion from bytes to string

* [All] Cleaned code

* Update README.md

* Fixed variable naming issue

* Fixed DataGenerator null pointer exception

* Sobstituded with correct conf file

* [v0.7.0] External Source Support: Kafka (#130)

* Added external Kafka Source with basic configurations (curr. only workflow schema)

* Fix: added control for shape and external source

* Added external Kafka Source also for Pipeline Schema

* Added debugging output in source

* cleaned code

* [v0.7.0] External Source Support: Kafka
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Conclusion: Solved The issue has been solved Platform: All The issue is related to all the applications Priority: Critical The issue is a blocker Type: Bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ale93p