Export Datstore to BigQuery #14

przemyslaw-jasinski · 2018-07-10T11:20:00Z

No description provided.

coveralls · 2018-07-10T11:23:11Z

Pull Request Test Coverage Report for Build 330

0 of 155 (0.0%) changed or added relevant lines in 5 files are covered.
37 unchanged lines in 3 files lost coverage.
Overall coverage decreased (-5.4%) to 82.396%

Changes Missing Coverage	Changed/Added Lines	%
src/appinfo.py	15	0.0%
src/datastore_export/export_datastore_to_big_query_service.py	18	0.0%
src/datastore_export/export_datastore_to_big_query_handler.py	31	0.0%
src/datastore_export/export_datastore_backups_to_gcs_service.py	43	0.0%
src/datastore_export/load_datastore_backups_to_big_query_service.py	48	0.0%

Files with Coverage Reduction	New Missed Lines	%
src/restore/test/table_randomizer.py	1	97.83%
src/main.py	5	0.0%
src/big_query/big_query.py	31	76.02%

Totals
Change from base Build 266:	-5.4%
Covered Lines:	2008
Relevant Lines:	2437

💛 - Coveralls

marcin-kolda · 2018-07-10T15:34:17Z

src/datastore_export/export_datastore_to_gcs_service.py

+        request = {
+            'project_id': app_id,
+            'output_url_prefix': output_url_prefix,
+            'entity_filter': entity_filter


entity_filter could be inlined

marcin-kolda · 2018-07-10T15:37:17Z

src/datastore_export/export_gcs_to_big_query_service.py

+        dataset_id, date = source_gcs_bucket.split("//")[1].split("/")
+        return {
+            "projectId": configuration.backup_project_id,
+            "location": "EU",


As discussed dataset and job should be created in the same region as GAE app.

marcin-kolda · 2018-07-10T15:38:59Z

src/datastore_export/export_datastore_to_big_query_handler.py

+    def get(self):
+        logging.info("Scheduling export of Datastore entities to GCS ...")
+        output_url = ExportDatastoreToGCSService\
+            .invoke(self.request, self.response)\


handler shouldn't pass request/response objects

marcin-kolda · 2018-07-10T15:39:39Z

src/datastore_export/export_datastore_to_big_query_handler.py

+        logging.info("Scheduling export of Datastore entities to GCS ...")
+        output_url = ExportDatastoreToGCSService\
+            .invoke(self.request, self.response)\
+            .wait_till_done(timeout=600)


could we simply raise an alert via error reporting if the whole request took more than 10 minutes?

marcin-kolda · 2018-07-10T15:40:49Z

src/datastore_export/export_datastore_to_gcs_service.py

+        app_id = configuration.backup_project_id
+        url = 'https://datastore.googleapis.com/v1/projects/%s:export' % app_id
+
+        output_url_prefix = cls.get_output_url_prefix(request)


We could use convention staging.project_id.appspot.com bucket, where lifecycle management is enabled. No need for configuration then

marcin-kolda · 2018-07-10T15:45:09Z

src/datastore_export/export_datastore_to_gcs_service.py

+class ExportDatastoreToGCSService(webapp2.RequestHandler):
+    @classmethod
+    def invoke(cls, request, response):
+        access_token, _ = app_identity.get_access_token(


Consider using google-api-python-client for consistency with other APIs

marcin-kolda · 2018-07-10T15:46:38Z

src/datastore_export/export_gcs_to_big_query_service.py

+        finish_time = time.time() + timeout
+        while time.time() < finish_time:
+            logging.info(
+                "Export from GCS to BQ - "


I would log load_job_id here as well

marcin-kolda · 2018-07-11T14:19:11Z

src/appinfo.py

+        return httplib2.Http(timeout=60)
+
+    @staticmethod
+    def __map(location_id):


Could we just check prefixes, not every zone? They change quite often now

marcin-kolda · 2018-07-11T14:19:26Z

src/appinfo.py

+            return "EU"
+        if location_id in ASIA_LOCATIONS:
+            return "Asia"
+        return "UNKNOWN"


What about Australia?

marcin-kolda · 2018-07-11T14:20:12Z

src/datastore_export/export_datastore_to_big_query_handler.py

+        ExportDatastoreBackupsToGCSService().export(gcs_output_uri, kinds)
+
+        logging.info("Loading Datastore backups from GCS to Big Query")
+        LoadDatastoreBackupsToBigQueryService(now_date)\


What happens here in case of timeout after error reporting is sent?

marcin-kolda · 2018-07-11T14:21:46Z

src/datastore_export/export_datastore_to_big_query_handler.py

+
+    @staticmethod
+    def __create_gcs_output_url(gcs_folder_name):
+        app_id = configuration.backup_project_id


This should be taken from current GAE app, backup project id could be different where we don't have access.

marcin-kolda · 2018-07-11T14:22:48Z

src/appinfo.py

+        if location_id in EU_LOCATIONS:
+            return "EU"
+        if location_id in ASIA_LOCATIONS:
+            return "Asia"


In Asia there is only: asia-northeast1
https://cloud.google.com/bigquery/docs/dataset-locations

marcin-kolda · 2018-07-11T14:24:02Z

src/datastore_export/load_datastore_backups_to_big_query_service.py

+    def __create_job_body(self, source_uri, kind):
+        return {
+            "projectId": configuration.backup_project_id,
+            "location": "EU",


EU?
Should be the same as dataset id I think

marcin-kolda · 2018-07-11T14:25:14Z

src/datastore_export/export_datastore_to_big_query_handler.py

+
+        self.response.headers['Content-Type'] = 'application/json'
+        self.response.set_status(200)
+        self.response.out.write(json.dumps({'status': 'success'}))


It's not success in case of timeout

radkomateusz · 2018-07-12T07:56:18Z

src/datastore_export/load_datastore_backups_to_big_query_service.py

+                "Loading Datastore backups from GCS to BQ (jobId: %s) - "
+                "waiting %d seconds for request to end...", load_job_id, PERIOD
+            )
+            time.sleep(PERIOD)


please move sleeping after check result logic to avoid waiting 1minute before checking for the first time as discussed

przemyslaw-jasinski and others added 16 commits July 6, 2018 11:08

YACHT-584: Created cron job that exports DS to GCS

c2c8415

YACHT-584: Added cloud datastore export to cron

5d6d37d

YACHT-584: Renamed file export_datastore_to_gcs

e2ee517

YACHT-584: Moved ds export cron from gitlab to github repo

5183f4e

export to bq tmp

6d5da50

YACHT-584: Trying to merge export of ds_to_gcs with gcs_to_bq

73bdc1e

YACHT-584: ExportDatastoreToGCS refactor

d896aac

YACHT-584: Changed export handler url in app.yaml

54a5c08

YACHT-584: export_ds_to_gcs now checks till finished

c243678

export to bq tmp part 2

852f027

YACHT-584: Merged exports from ds to gcs with gcs to bq

920dcab

YACHT-584: Fixed bugs

f41ded9

YACHT-584: Fixed date format

60e1c1c

YACHT-584: Refactor

ce9cd2f

YACHT-584: Fixed some small bug

fec3577

YACHT-584: Changed tables suffix from datetime to date

6d4f331

przemyslaw-jasinski requested review from marcin-kolda and radkomateusz July 10, 2018 11:20

marcin-kolda suggested changes Jul 10, 2018

View reviewed changes

przemyslaw-jasinski added 7 commits July 10, 2018 18:59

YACHT-584: Trying to replace invoking url with datastore python client

146225a

YACHT-584: Fixed usage of datastore client

6295b88

YACHT-584: Refactor

c0c2404

YACHT-584: Now service creates dataset in default location if not exists

ed773b5

YACHT-584: Added handler to cron

b99f280

YACHT-584: Fixed cron schedule and retry parameters

783afdb

YACHT-584: Cron yaml refactor

87b190c

marcin-kolda suggested changes Jul 11, 2018

View reviewed changes

przemyslaw-jasinski added 2 commits July 11, 2018 17:12

YACHT-584: refactor

80ddf41

YACHT-584: Refactor

ce111e9

radkomateusz suggested changes Jul 12, 2018

View reviewed changes

przemyslaw-jasinski added 3 commits July 12, 2018 10:05

YACHT-584: Refactor

d1d92aa

YACHT-584: Refactor

b5a5471

YACHT-584: Fixed bug

46d5f07

radkomateusz previously approved these changes Jul 12, 2018

View reviewed changes

YACHT-584: Fixed small bug

bf60986

przemyslaw-jasinski dismissed radkomateusz’s stale review via bf60986 July 12, 2018 09:06

radkomateusz approved these changes Jul 12, 2018

View reviewed changes

przemyslaw-jasinski merged commit cebbdef into master Jul 12, 2018

przemyslaw-jasinski deleted the export_ds_to_bq branch July 12, 2018 10:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Export Datstore to BigQuery #14

Export Datstore to BigQuery #14

przemyslaw-jasinski commented Jul 10, 2018

coveralls commented Jul 10, 2018 •

edited

Loading

marcin-kolda Jul 10, 2018

marcin-kolda Jul 10, 2018 •

edited

Loading

marcin-kolda Jul 10, 2018

marcin-kolda Jul 10, 2018

marcin-kolda Jul 10, 2018

marcin-kolda Jul 10, 2018

marcin-kolda Jul 10, 2018

marcin-kolda Jul 11, 2018

marcin-kolda Jul 11, 2018

marcin-kolda Jul 11, 2018

marcin-kolda Jul 11, 2018

marcin-kolda Jul 11, 2018

marcin-kolda Jul 11, 2018

marcin-kolda Jul 11, 2018

radkomateusz Jul 12, 2018

Export Datstore to BigQuery #14

Export Datstore to BigQuery #14

Conversation

przemyslaw-jasinski commented Jul 10, 2018

coveralls commented Jul 10, 2018 • edited Loading

Pull Request Test Coverage Report for Build 330

💛 - Coveralls

Choose a reason for hiding this comment

marcin-kolda Jul 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Jul 10, 2018 •

edited

Loading

marcin-kolda Jul 10, 2018 •

edited

Loading