Avoid memory problems while exporting big datasets to excel #6403
Comments
metas-ts
added a commit
that referenced
this issue
Mar 26, 2020
* change AbstractExcelExporter such that it doesn't require random-access to the underlying data source: * basically this is achieved by replacing `getValueAt(int row, int col)` with `getNextRow()` * it only iterated the data-source one-by-one anyways * Define a DataConsumer interface that can be passed as parameter to ExcelExporterService, such that the jdbc result-set can be added to it * Implement DataConsumer by JdbcExcelExporter which also subclasses AbstractExcelExporter * that way we can pass JdbcExcelExporter to ExcelExporterService and have it push the whole resultset to excel. * Move code from JdbcExportDataSource to DB such that we can have a fetch-sized resultset and thus avoid loading all the data from the DB at once #6403
metas-ts
added a commit
that referenced
this issue
Mar 26, 2020
fixes regarding header row creation and multiple sheets #6403
metas-ts
added a commit
that referenced
this issue
Mar 27, 2020
metas-ts
added a commit
to metasfresh/metasfresh-webui-api-legacy
that referenced
this issue
Mar 27, 2020
metas-ts
added a commit
that referenced
this issue
Mar 27, 2020
* Avoid memory problems while exporting big datasets to excel * change AbstractExcelExporter such that it doesn't require random-access to the underlying data source: * basically this is achieved by replacing `getValueAt(int row, int col)` with `getNextRow()` * it only iterated the data-source one-by-one anyways * Define a DataConsumer interface that can be passed as parameter to ExcelExporterService, such that the jdbc result-set can be added to it * Implement DataConsumer by JdbcExcelExporter which also subclasses AbstractExcelExporter * that way we can pass JdbcExcelExporter to ExcelExporterService and have it push the whole resultset to excel. * Move code from JdbcExportDataSource to DB such that we can have a fetch-sized resultset and thus avoid loading all the data from the DB at once #6403 * Avoid memory problems while exporting big datasets to excel foolow codacy warnings #6403 * Avoid memory problems while exporting big datasets to excel fixes regarding header row creation and multiple sheets #6403 * Avoid memory problems while exporting big datasets to excel minor log improvements #6403 * minor file name change to indicate the intention * Avoid memory problems while exporting big datasets to excel after-review-fixes #6403 * Avoid memory problems while exporting big datasets to excel minor #6403 (cherry picked from commit 53efc43) solved Conflicts: de.metas.adempiere.adempiere/base/src/main/java-legacy/de/metas/impexp/excel/service/ExcelExporterService.java de.metas.adempiere.adempiere/base/src/main/java-legacy/org/compiere/util/DB.java de.metas.adempiere.adempiere/migration/src/main/sql/postgresql/system/10-de.metas.adempiere/5551809_sys_AD_Process_IsNotifyUserAfterExecution_DDL.sql
metas-ts
added a commit
to metasfresh/metasfresh-webui-api-legacy
that referenced
this issue
Mar 27, 2020
metasfresh/metasfresh#6403 (cherry picked from commit 31c0828)
metas-ts
added a commit
to metasfresh/metasfresh-webui-api-legacy
that referenced
this issue
Mar 27, 2020
metasfresh/metasfresh#6403 (cherry picked from commit 31c0828)
metas-ts
added a commit
that referenced
this issue
Mar 27, 2020
* Avoid memory problems while exporting big datasets to excel * change AbstractExcelExporter such that it doesn't require random-access to the underlying data source: * basically this is achieved by replacing `getValueAt(int row, int col)` with `getNextRow()` * it only iterated the data-source one-by-one anyways * Define a DataConsumer interface that can be passed as parameter to ExcelExporterService, such that the jdbc result-set can be added to it * Implement DataConsumer by JdbcExcelExporter which also subclasses AbstractExcelExporter * that way we can pass JdbcExcelExporter to ExcelExporterService and have it push the whole resultset to excel. * Move code from JdbcExportDataSource to DB such that we can have a fetch-sized resultset and thus avoid loading all the data from the DB at once #6403 * Avoid memory problems while exporting big datasets to excel foolow codacy warnings #6403 * Avoid memory problems while exporting big datasets to excel fixes regarding header row creation and multiple sheets #6403 * Avoid memory problems while exporting big datasets to excel minor log improvements #6403 * minor file name change to indicate the intention * Avoid memory problems while exporting big datasets to excel after-review-fixes #6403 * Avoid memory problems while exporting big datasets to excel minor #6403 (cherry picked from commit 53efc43) solved Conflicts: de.metas.adempiere.adempiere/base/src/main/java-legacy/de/metas/impexp/excel/service/ExcelExporterService.java de.metas.adempiere.adempiere/base/src/main/java-legacy/org/compiere/util/DB.java de.metas.adempiere.adempiere/migration/src/main/sql/postgresql/system/10-de.metas.adempiere/5551809_sys_AD_Process_IsNotifyUserAfterExecution_DDL.sql de.metas.marketing/cleverreach/src/test/java/de/metas/marketing/gateway/cleverreach/ManualTest.java
metas-ts
added a commit
that referenced
this issue
Mar 28, 2020
* [#6403](#6403) Avoid memory problems while exporting big datasets to excel * [#6393](#6393) Generating delivery days for >1 tour not working * [#1394](metasfresh/metasfresh-webui-api-legacy#1394) layout element: provide maxLength for Text and LongText fields * [#6364](#6364) Fixes/ adjustments: Excel Sales Order adjustments * [#6376](#6376) Avoid empty text for packing material in Product specifications * [#6382](#6382) Identifiers are swapped for Requests and Invoices * [#6390](#6390) Fix User Queries with dates throwing error * [#6405](#6405) Shipment dispo: cache invalidate when mass updating the records * [#2556](metasfresh/metasfresh-webui-frontend-legacy#2556) make sure the view filters are displayed exactly in the same order we have them in the layout
metas-ts
added a commit
that referenced
this issue
Mar 30, 2020
metas-ts
added a commit
that referenced
this issue
Apr 22, 2020
* fix bug where connection was not returned #6403 (cherry picked from commit 6f2fb1a) * solve commission related problems (#6424) * CommissionShareHandler: solve #6299 for non-item products * note that by default the commission product is in fact such a non-item (i.e. service) product * 5556000_sys_gh6421_fix_c_flatrate_condition_mandatory_problem.sql: solve #6421 by updating the AD_Column's mandatory logic * OrderCheckupBuilder: minor unrelated (cherry picked from commit ad11efc) * WIP - add different EPs - unit tests currently fail #6459 * WIP - clean up a bit and fix tests #6459 * 6459-app - add beneficiary columns to invoice and.. ..code to extract it from healtcare-XML also improve logging #6459 * 6459-app - add beneficiary columns to invoice and.. ..do a little bit of cleanup also improve logging #6459 * 6459-app - Allow healtchcare-ch EA invoice * add attaachment listener to extract beneficiary-bpartner-id from XML into invoice * pimp AttachmentListener invocation; it is now called whenever am attachment is linked to a record * fix different problem around UPDATE_REMOVE with bpartner-location and contacts * fix problem regarding BPartner-creation if current user's org differs from the JSON's target-org #6459 * fix unit tests #6459 * Solve bpartner lookup problem when getCreating org master data (#6492) #6491 * minor changes * Solve bpartner lookup problem when getCreating org master data fix unit tests #6491 * Solve bpartner lookup problem when getCreating org master data - add property to olcand-JSON that allows to advise metasfresh on the way to lookup existing bpartner data (GLN, externalId or value) - cleanup in JsonServiceFactory: remove obsolete RecordChangeLogRepository stuff - improve logging #6491 * fix unit tests #6491 * 6493-app - Distinguish betwenn null and not set in BPartner-API (#6494) * Solve bpartner lookup problem when getCreating org master data (#6492) #6491 * minor changes * Solve bpartner lookup problem when getCreating org master data fix unit tests #6491 * 6493-app - Distinguish betwenn null and not set in BPartner-API WIP - start experimenting with bpartner-contact; many thanks to https://stackoverflow.com/a/48436906/1012103 ! #6493 * Solve bpartner lookup problem when getCreating org master data - add property to olcand-JSON that allows to advise metasfresh on the way to lookup existing bpartner data (GLN, externalId or value) - cleanup in JsonServiceFactory: remove obsolete RecordChangeLogRepository stuff - improve logging #6491 * fix unit tests #6491 * 6493-app - Distinguish betwenn null and not set in BPartner-API WIP - i think the bulk of the code changes is done, but need much more QA #6493 * unrelated logging improvement * 6493-app - Distinguish betwenn null and not set in BPartner-API testing #6493 * 6493-app - Distinguish betwenn null and not set in BPartner-API testing #6493 * unrelated logging improvement - fix unit test * fix cornercase bug: if e.g. a location upsert request item was identified by ext-123 and the location's externalId was changed in that request item, then metasfresh lost track of the item, resulting in a nasty exception. this commit fixes that problem * Allow healtchcare-ch EA invoice #6459 - change POReferenceAsSequenceNoProvider and surrounding code for the document number to be suffixed by a "regular" incremental seqno - solve some after-merge compile errors * Allow healtchcare-ch EA invoice #6459 - fix bug regarding different patient and guarantor address - add normal formatting to healthcare documentNo AD_Sequences * change currentnext for healthcare-ch #6459 * Update PageNotFoundException.java * Update JsonResponseBPartner.java * Update pom.xml cleanup * minor #6459 * fix unit tests #6459 * remove update to field-length..prevent problems on custmized instances
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is this a bug or feature request?
FR
What is the current behavior?
Which are the steps to reproduce?
What is the expected or desired behavior?
The text was updated successfully, but these errors were encountered: