Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

split data export archive paths on provider uuid #1352

Merged
merged 16 commits into from
Nov 7, 2019
Merged

split data export archive paths on provider uuid #1352

merged 16 commits into from
Nov 7, 2019

Conversation

infinitewarp
Copy link
Contributor

for #1315

@codecov
Copy link

codecov bot commented Nov 4, 2019 •
edited

Codecov Report

❗ No coverage uploaded for pull request base (master@3992405). Click here to learn what that means.
The diff coverage is 100%.

@@           Coverage Diff            @@
##             master   #1352   +/-   ##
========================================
  Coverage          ?   95.9%           
========================================
  Files             ?     200           
  Lines             ?   11246           
  Branches          ?    1257           
========================================
  Hits              ?   10783           
  Misses            ?     290           
  Partials          ?     173

@infinitewarp infinitewarp force-pushed the 1315-archive-provider-id branch 6 times, most recently from b3fb239 to c641157 Compare November 5, 2019 16:11
@infinitewarp infinitewarp requested a review from a team November 5, 2019 16:20
katherine-black
katherine-black previously approved these changes Nov 5, 2019
View reviewed changes
werwty
werwty previously approved these changes Nov 5, 2019
View reviewed changes
werwty
werwty reviewed Nov 5, 2019
View reviewed changes
koku/masu/util/upload.py Outdated Show resolved Hide resolved
@infinitewarp infinitewarp force-pushed the 1315-archive-provider-id branch 3 times, most recently from 44f9f78 to 3025fae Compare November 7, 2019 19:01
@infinitewarp
separate data exports by provider uuid
1d43f04
@infinitewarp
don't reuse the same cursor for all queries, and don't return early i…
3373a1c 
…f any date has no results
@infinitewarp
don't hold the db cursor while performing the file upload
7585bc2
@infinitewarp
sync data exports in s3 with paths separated by provider uuid
b6b422e
@infinitewarp
move table export settings to supporting module
5f297d4
@infinitewarp
test all table_export_settings, not just one
4699d1b
@infinitewarp
D401 First line should be in imperative mood; rephrasing
f9a95b6
@infinitewarp
lock "today" in test_upload to an arbitrary date so we are unaffected…
bd7cae7 
… by when the test actually runs
@infinitewarp
use "schema_name" more consistently as a variable name
4f01ee9
@infinitewarp
add upload and customer_data_sync queues to docker-compose koku celer…
2d82419 
…y worker
@infinitewarp
read from env in docker-compose for ENABLE_S3_ARCHIVING, S3_BUCKET_NA…
7282e85 
…ME, and S3_BUCKET_PATH
@infinitewarp
reduce this log from info to debug
0c1579e
@infinitewarp
make queries correctly reference the ocp provider uuids
200aa58
@infinitewarp infinitewarp changed the title WIP split data export archive paths on provider uuid split data export archive paths on provider uuid Nov 7, 2019
@infinitewarp infinitewarp marked this pull request as ready for review November 7, 2019 20:26
@infinitewarp
Copy link
Contributor Author

demo of current functionality: https://asciinema.org/a/279862

werwty
werwty reviewed Nov 7, 2019
View reviewed changes
koku/api/dataexport/syncer/__init__.py Outdated Show resolved Hide resolved
koku/masu/celery/tasks.py Outdated Show resolved Hide resolved
@codeclimate
Copy link

codeclimate bot commented Nov 7, 2019

Code Climate has analyzed commit 6158d70 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (90% is the threshold).

This pull request will bring the total coverage in the repository to 97.4%.

View more on Code Climate.

@infinitewarp infinitewarp merged commit 17cef16 into master Nov 7, 2019
@infinitewarp infinitewarp deleted the 1315-archive-provider-id branch November 7, 2019 21:19
infinitewarp added a commit to infinitewarp/demo-notes that referenced this pull request Nov 18, 2019
@infinitewarp
koku split archive by provider
5520764 
https://asciinema.org/a/279862
https://asciinema.org/a/279858
project-koku/koku#1352
https://github.com/project-koku/koku/issues/1315
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@werwty werwty werwty approved these changes

@katherine-black katherine-black katherine-black left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@infinitewarp @werwty @katherine-black