Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add cleanup listener #159

Closed
morazow opened this issue May 22, 2023 · 1 comment
Closed

Add cleanup listener #159

morazow opened this issue May 22, 2023 · 1 comment
Assignees
@morazow
Labels
feature Product feature

Comments

@morazow
Copy link
Contributor

morazow commented May 22, 2023

Situation

We are using an intermediate storage layer to exchange data between Spark and Exasol. We should add an Spark listener so that the intermediate data is deleted once the Spark job is finalized.

Acceptance Criteria

  • A Spark listener added to be run at the end of a Spark job to clean up the intermediate location
@morazow morazow added the feature Product feature label May 22, 2023
morazow added a commit that referenced this issue May 24, 2023
@morazow
Merge branch 'feature/#160-write-to-exasol' into feature/#159-cleanup
df1f5d6
morazow added a commit that referenced this issue May 24, 2023
@morazow
Added initial version of cleanup listener
65bee47 
Fixes #159
morazow added a commit that referenced this issue May 25, 2023
@morazow
Merge branch 'feature/#160-write-to-exasol' into feature/#159-cleanup
8add77f
morazow added a commit that referenced this issue May 25, 2023
@morazow
Merge branch 'develop' into feature/#159-cleanup
f0df881
@morazow morazow mentioned this issue May 25, 2023
Merged
@ckunki
Copy link
Contributor

ckunki commented May 26, 2023

my effort for review was ~ .5 days

morazow added a commit that referenced this issue May 31, 2023
@morazow @ckunki
Feature/#159 cleanup ck proposal (#166)
6c3d49f 
* Refactored tests inS3CleanupIT

* improved and simplified solution

---------

Co-authored-by: KK <christoph.kuhnke@exasol.com>
morazow added a commit that referenced this issue May 31, 2023
@morazow @ckunki
#159: Added cleanup process to remove intermediate data after job fin…
4cf041a 
…ish (#165)

Fixes #159

---------

Co-authored-by: Christoph Kuhnke <github@kuhnke.net>
Co-authored-by: KK <christoph.kuhnke@exasol.com>
@morazow morazow closed this as completed May 31, 2023
morazow added a commit that referenced this issue Jul 14, 2023
@morazow @kaklakariada
@ckunki @tkilias @redcatbear
Prepare for release of 2.0.0 (#186)
e522e56 
* #150: Added S3 intermediate storage layer (#152)


Co-authored-by: Christoph Pirkl <christoph.pirkl@exasol.com>

* #149: Added s3 import query generator and runner (#156)


Fixes #149

Co-authored-by: Christoph Kuhnke <github@kuhnke.net>
Co-authored-by: KK <christoph.kuhnke@exasol.com>

* #160: Add support for writing to Exasol database using S3 as intermediate storage (#161)


Fixes #160

* Address review suggestion from @tkilias

* #159: Added cleanup process to remove intermediate data after job finish (#165)

Fixes #159

---------

Co-authored-by: Christoph Kuhnke <github@kuhnke.net>
Co-authored-by: KK <christoph.kuhnke@exasol.com>

* #158: Refactored options class (#167)

Fixes #158

---------

Co-authored-by: Torsten Kilias <tkilias@users.noreply.github.com>

* #168: Refactored to add module setup (#170)


Fixes #168

* #171: Refactored artifact packaging and releasing process for module setup (#173)


---------

Co-authored-by: Sebastian Bär <sebastian.baer@exasol.com>

* #174: Refactored GitHub workflow actions (#175)

Fixes #175

* #176: Fixed sha256sum files upload bug (#179)

* #155: Refactored to unify user parameters (#178)

Fixes #155

---------

Co-authored-by: Christoph Pirkl <christoph.pirkl@exasol.com>

* #164: Added validation check before writing to intermediate location (#180)

Fixes #164

---------

Co-authored-by: Christoph Pirkl <christoph.pirkl@exasol.com>

* #162: Updated developer guide with staged commit behaviour (#182)

Fixes #162

---------

Co-authored-by: Christoph Pirkl <christoph.pirkl@exasol.com>

* #183: Updated user guide to include latest changes (#184)

Fixes #183

---------

Co-authored-by: Christoph Pirkl <christoph.pirkl@exasol.com>

* Updated version to `2.0.0` (#185)

* Changed version
* Improved default options
* Updated changes and user guide docs

---------

Co-authored-by: Christoph Pirkl <christoph.pirkl@exasol.com>
Co-authored-by: Christoph Kuhnke <github@kuhnke.net>
Co-authored-by: KK <christoph.kuhnke@exasol.com>
Co-authored-by: Torsten Kilias <tkilias@users.noreply.github.com>
Co-authored-by: Sebastian Bär <sebastian.baer@exasol.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@morazow morazow

Labels
feature Product feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@morazow @ckunki