Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clear downloads #228

Merged
merged 5 commits into from
Jan 29, 2019
Merged

Clear downloads #228

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
217fea9
Implement pattern for deleting contents of downloads
reginafcompton Jan 28, 2019
ecc39e7
Delete everything at the endof import
reginafcompton Jan 29, 2019
a15be26
Simplify with keep_downloads arg
reginafcompton Jan 29, 2019
eb8311b
Peg flake8 to a version, and ignore W504
reginafcompton Jan 29, 2019
2f155d8
Better comment
reginafcompton Jan 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions councilmatic_core/management/commands/import_data.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import sys
import logging
import logging.config
import shutil

import requests
import pytz
Expand Down Expand Up @@ -96,6 +97,10 @@ def add_arguments(self, parser):
default=False,
help='Only download OCD data')

parser.add_argument('--clear_downloads',
default=True,
help='Clear downloads after successful import')

def handle(self, *args, **options):

self.connection = engine.connect()
Expand Down Expand Up @@ -134,6 +139,7 @@ def handle(self, *args, **options):

download_only = options['download_only']
import_only = options['import_only']
clear_downloads = options['clear_downloads']

if not import_only and not download_only:
download_only = True
Expand All @@ -143,6 +149,7 @@ def handle(self, *args, **options):
etl_method = getattr(self, '{}_etl'.format(endpoint))
etl_method(import_only=import_only,
download_only=download_only,
clear_downloads=clear_downloads,
delete=options['delete'])

except Exception as e:
Expand Down Expand Up @@ -181,6 +188,7 @@ def log_message(self,
def organizations_etl(self,
import_only=True,
download_only=True,
clear_downloads=True,
delete=False):

if download_only:
Expand All @@ -204,6 +212,9 @@ def organizations_etl(self,
self.add_new_organizations()
self.add_new_posts()

if clear_downloads:
shutil.rmtree(self.organizations_folder)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reginafcompton Are you planning to add this to the other download folders, too?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to this question, otherwise looks regular to me!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, i just saw your comment in the description! fire away with the other folders.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all right - the way it should work: by default, import_data removes all files from all download directories. @evz - I think we should go folder-by-folder, rather than do a "global" remove-everything-everytime, in the event that an import fails and I need the downloads for debugging (or I simply do not want to wait for the script to download everything again).

@hancush - ready aim fire

self.log_message('Organizations Complete!',
fancy=True,
center=True,
Expand Down