GPII-3138: Move functionality of gpii-dataloader repo into universal #692

klown · 2018-10-18T20:52:29Z

@stepanstipl @cindyli @mrtyler, Here is a first pass at moving the gist of what gpii-dataloader pull 6. does into universal. I'm making separate pull request from pull 626 to avoid confusion. If that is the correct place to put it, I can move easily.

It's not complete, but good enough for others to start commenting. In particular, the relevant parts of the README need to come over, and I haven't start that yet.

For more background, see this discussion on gpii-dataloader pull 6.

In addition, @stepanstipl created a similar pull request (#696) which has since been merged into this one. Here is a copy of his PR description:

This PR moves dataloader from original gpii-ops/gpii-dataloader repo into universal.

Main reasons behind this:

Get rid of the duplication of Dockerfile and necessity to maintain extra image and pipeline
Get rid of cloning the repo and installing npm modules when the container is executed (this would be slow, is prone to errors and would raise question of versioning)
Data loader will always have the new universal code (as it will be running in the universal image)
Data loader will get triggered every time there's a change to universal (as the universal image will get updated)

Most of the code was in universal already, and gpii-ops/gpii-dataloader contained basically only shell wrapper and the docker image. Work on new dataloader (GPII-3138 and #626) requires changes to the existing dataloader wrapper and therefore presented a good opportunity for the move.

This is a continuation of gpii-ops/gpii-dataloader#6 PR.

Main changes:

Import deleteAndLoadSnapsets.sh and DataLoader.md
Add dataLoader dependencies to docker image (jq and curl)
Update data loader script to reflect new location in universal
Minor formatting changes
Add DB connectivity check to dataloader
Update vagrantCloudBasedContainers.sh to work with new dl location
Update DataLoader docs to reflect current state

Instead of building and running a docker container, move the relevant code into universal

gpii-bot · 2018-10-18T21:25:49Z

CI job passed: https://ci.gpii.net/job/universal-tests/1206/

klown · 2018-10-18T21:36:03Z

CI job passed: https://ci.gpii.net/job/universal-tests/1206/

Even though it passed, I discovered that jq was not found. It's needed for the warm_indices() function. Buried in the console log for the production tests:

/home/vagrant/sync/universal/scripts/deleteAndLoadSnapsets.sh: line 14: jq: command not found

I guess CI doesn't include it?

…inal slash

…rsal docker image

… data loader

Added check for the required number of command line arguments.

GPII-3138: Merged Dataloader code import from Stepan

stepanstipl · 2018-10-22T18:48:55Z

Thank @klown for merging in my work. I'll just paste my original PR comment below as it adds a bit of context (can't update this PR's description):

This PR moves dataloader from original gpii-ops/gpii-dataloader repo into universal.

Main reasons behind this:

Get rid of the duplication of Dockerfile and necessity to maintain extra image and pipeline
Get rid of cloning the repo and installing npm modules when the container is executed (this would be slow, is prone to errors and would raise question of versioning)
Data loader will always have the new universal code (as it will be running in the universal image)
Data loader will get triggered every time there's a change to universal (as the universal image will get updated)

Most of the code was in universal already, and gpii-ops/gpii-dataloader contained basically only shell wrapper and the docker image. Work on new dataloader (GPII-3138 and #626) requires changes to the existing dataloader wrapper and therefore presented a good opportunity for the move.

This is a continuation of gpii-ops/gpii-dataloader#6 PR.

Main changes:

Import deleteAndLoadSnapsets.sh and DataLoader.md
Add dataLoader dependencies to docker image (jq and curl)
Update data loader script to reflect new location in universal
Minor formatting changes
Add DB connectivity check to dataloader
Update vagrantCloudBasedContainers.sh to work with new dl location
Update DataLoader docs to reflect current state

klown · 2018-10-22T19:07:24Z

Thank @klown for merging in my work. I'll just paste my original PR comment below as it adds a bit of context (can't update this PR's description) ...

No problem @stepanstipl . Well, I can edit this PR's description, so I copied/promoted the gist of it.

Fixed minor problems from previous merge of Stepan's pull request.

klown · 2018-10-22T20:56:48Z

@stepanstipl I cleaned up the merge issues -- not too many in the end. But a second pair of eyes wouldn't hurt.

gpii-bot · 2018-10-22T22:09:43Z

CI job passed: https://ci.gpii.net/job/universal-tests/1242/

gpii-bot · 2018-10-22T22:40:23Z

CI job failed: https://ci.gpii.net/job/universal-tests/1243/

gpii-bot · 2018-10-22T23:11:09Z

CI job passed: https://ci.gpii.net/job/universal-tests/1244/

amb26 · 2018-10-22T23:52:10Z

documentation/DataLoader.md

+
+- Converts the preferences in universal into `snapset` Prefs Safes and GPII Keys,
+- Optionally deletes existing database,
+- Creates a CouchDB database if none exits,


Typo exists

and it's "exists" :-)
Fixed.

Never mind -- merging in Stepan's pull

amb26 · 2018-10-22T23:53:08Z

documentation/DataLoader.md

+
+## Environment Variables
+
+- `COUCHDB_URL`: URL of the CouchDB database. (required)


Can we add prefixes to these (probably GPII_) to reduce chances of conflicts with other uses of the environment (and also as a hint/courtesy to anyone looking in the environment wondering how they got there)

Yes, that sounds like a good idea (although the chance of conflict is pretty much non-existent - data loader runs in its own container and there's nothing else expected to be running).

amb26 · 2018-10-22T23:56:02Z

documentation/DataLoader.md

+It does following:
+
+- Converts the preferences in universal into `snapset` Prefs Safes and GPII Keys,
+- Optionally deletes existing database,


Puzzled that the option here is to delete the entire database rather than merely all of the documents of type "snapset"

There is a production test (test:vagrantProduction) that executes vagrantCloudBasedContainers.sh without its --no-rebuild flag. When first run locally, or when run by CI, there is no database to delete, so deleting the database is irrelevant at that point.

But, if a developer runs the test a second time, they can at their option start from scratch with an empty database, or use the --no-rebuild flag and modify the database in situ, which is closer to a production environment.

The question is whether the first option -- start from scratch -- is useful.

OK, I think this is just a documentation issue, since if I understand correctly, the action of steps 5 and 6 is actually the one of https://github.com/GPII/universal/blob/master/scripts/deleteAndLoadSnapsets.js - so we should just clarify the wording in the readme here to explain that steps 5 and 6 don't simply drop "snapsets and keys" but in particular only those keys which are associated with snapsets. I think it would be helpful for the comment here to explicitly link to or reproduce the comment at the head of the script https://github.com/GPII/universal/blob/master/scripts/deleteAndLoadSnapsets.js#L11 so that, for example, anyone invoking this script will do so in confidence that it will not delete user data (unless they enable GPII_CLEAR_INDEX, which should be supplied with a clear warning)

OK, I've added to the README -- take a look.

stepanstipl · 2018-10-24T15:40:48Z

BTW, I'm also updating the dataloader's README as @amb26 suggested. I hope I'm not stepping on your toes here.

Ok great, I'm not doing anything about that one :).

gpii-bot · 2018-10-24T16:52:13Z

CI job passed: https://ci.gpii.net/job/universal-tests/1257/

- Called out that only views, and snapset PrefsSafes and their associated GPII keys are deleted/updated (steps 4, 5, and 6), - Explained usage of environment variables, - Added a warning about GPII_CLEAR_INDEX, - Explained usage difference between development vs. staging/production environments.

gpii-bot · 2018-10-24T20:38:48Z

CI job failed: https://ci.gpii.net/job/universal-tests/1262/

klown · 2018-10-24T20:48:37Z

CI job failed: https://ci.gpii.net/job/universal-tests/1262/

It can't even create the VM: https://ci.gpii.net/job/universal-create-vm/782/console

Trying again...

klown · 2018-10-24T20:48:45Z

ok to test

gpii-bot · 2018-10-24T21:21:49Z

CI job failed: https://ci.gpii.net/job/universal-tests/1263/

stepanstipl · 2018-10-25T08:54:37Z

it's the very same jq: Test concluded - Module "Testing Pouch Manager" Test name "Testing the load and the reset processes": 21/22 passed - FAIL causing this issue.

It seems to be quite common reason for failure (just a quick search - #638, #632, #534 and those are only the fails that actually copy-pasted the fail message).

I've started looking if I can reproduce this faling test locally & do something about it, but I'm not very familiar with the code and seems quite complex at a first look.

amb26 · 2018-10-25T10:15:19Z

Thanks for investigating, @stepanstipl - actually I think that @the-t-in-rtf is inclining to believe that, as a result of failures like this one, we should abandon the use of PouchDB for our integration tests since it seems prone to erratic failures like this one. @the-t-in-rtf - does any reasonably straightforward means strike you for reducing the probability of these failures in the meantime?

the-t-in-rtf · 2018-10-25T11:46:52Z

@amb26, my first suspicion is that we would see this less often if we weren't reusing a single data directory between runs, as configured here:

https://github.com/GPII/universal/blob/master/gpii/node_modules/pouchManager/test/pouchManagerTests.js#L257

My suggestion would be to update that to include unique information like {that}.id, so that there is no chance of cleanup failures in earlier tests polluting the next run.

stepanstipl · 2018-10-25T13:09:00Z

Thanks @amb26 and @the-t-in-rtf

I completely failed to reproduce the issue locally - even re-running the pouchManager tests 1000x times in a loop, and running multiple of those in parallel, I wasn't able to get that error. Even messing with the files in /tmp/pouchManagerTests directory during the test run (creating extra or deleting the LOCK and other files), I still wasn't able to get the same error, so I don't really know what's going on.

Adding a variable part to the dir name sounds like a good idea - I've done it in klown#6 against this branch - but as I wasn't able to reproduce the error locally, can't say if it really helps :D.

GPII-3138: Add unique ID to pouchManager temp test dir

gpii-bot · 2018-10-25T13:56:47Z

CI job passed: https://ci.gpii.net/job/universal-tests/1264/

stepanstipl · 2018-10-25T14:06:03Z

ok to test

klown · 2018-10-25T14:06:17Z

@stepanstipl I've pulled in the variable directory name change, ran it locally, and all tests passed. They passed in CI as well.

But, as you noted, I don't know if this fixes the issue.

Adding a variable part to the dir name sounds like a good idea - I've done it in klown#6 against this branch - but as I wasn't able to reproduce the error locally, can't say if it really helps :D.

stepanstipl · 2018-10-25T14:08:38Z

@klown thanks for that, I just noticed it passed the CI. I was just trying to trigger re-run to try to get a better idea if it's just a random success or if it might have helped :D

But I don't think my comment triggered anything.

klown · 2018-10-25T14:13:53Z

@stepanstipl Good idea to run the test a couple of times 👍

On a related note, I'm not sure you have the authority to trigger CI -- nothing seems to be happening. @amatas can we add @stepanstipl to the group, if he isn't a member?

the-t-in-rtf · 2018-10-25T14:33:49Z

In the mean time, ok to test.

gpii-bot · 2018-10-25T15:06:32Z

CI job passed: https://ci.gpii.net/job/universal-tests/1266/

klown · 2018-10-25T15:08:41Z

Third time's the charm, ok to test.

gpii-bot · 2018-10-25T16:12:02Z

CI job passed: https://ci.gpii.net/job/universal-tests/1268/

stepanstipl · 2018-10-25T16:32:26Z

@klown ok, that looks optimistic 🤞... is there anything else to be done before we can merge this?

klown · 2018-10-25T16:52:26Z

Thanks @amb26 and congrats @stepanstipl

the-t-in-rtf · 2018-10-26T09:52:50Z

So, just as an afternote, In viewing all the rimraf cleanup errors in the logs and remembering past problems, my theory is that rimraf doesn't consistently complete its cleanup on particular platforms, including the one that's used to test in CI, and that this results in dirty data directories when the same location is used over and over again in different test runs. Even though I am thinking of replacing express-pouchdb, it could still be relevant when working directly with PouchDB instances outside of express, as the same "cleanup promise with timeout" is used there.

If it happens before we have an alternative approach, another option when working with express-pouchdb is to increase this timeout.

the-t-in-rtf · 2018-10-26T09:59:33Z

Can @amb26 or someone else with permission cut a dev release of universal that includes this work?

amb26 · 2018-10-26T10:38:09Z

0.3.0-dev.20181026T103556Z.112fa057

the-t-in-rtf · 2018-10-26T12:45:19Z

Cheers!

GPII-3138: Moved parts of gpii-dataloder code into universal

d5562d5

Instead of building and running a docker container, move the relevant code into universal

stepanstipl added 11 commits October 22, 2018 17:36

GPII-3138: Import deleteAndLoadSnapsets.sh and DataLoader.md

9e5346c

GPII-3138: Add dataLoader dependencies to docker image (jq and curl)

5c28281

GPII-3138: Modify convertPrefs.js to expect directories without the f…

5f3921a

…inal slash

GPII-3138: Update data loader script to reflect new location in unive…

044aa10

…rsal docker image

GPII-3138: Update directory checks to match containerised environment

d59bf09

GPII-3138: Minor formatting changes

54d75f7

GPII-3138: Add DB connectivity check to dataloader

68a961b

GPII-3138: Update vagrantCloudBasedContainers.sh to wor with omported…

fb3d435

… data loader

GPII-3138: Update DataLoader docs to reflect current state

26995c2

GPII-3138: Fix linting errors on DataLoader.md

d44b584

GPII-3138: Revert unintended change in DataLoader readme

07ace4c

stepanstipl mentioned this pull request Oct 22, 2018

Dataloader import klown/universal#3

Merged

GPII-3138: Improved check for proper usage of convertPrefs.js

eb92c4e

Added check for the required number of command line arguments.

stepanstipl mentioned this pull request Oct 22, 2018

Dataloader move to universal from gpii-ops/gpii-dataloader #696

Closed

Merge pull request #3 from stepanstipl/dataloader-import-klown

2d8abe9

GPII-3138: Merged Dataloader code import from Stepan

This was referenced Oct 22, 2018

GPII-3391, GPII-3138: New dataloader in universal gpii-ops/gpii-version-updater#10

Merged

GPII-3391: New dataloader (orig: Add force_update to gpii-dataloader) gpii-ops/gpii-infra#163

Merged

GPII-3138: Fixed merge issues.

f28c79a

Fixed minor problems from previous merge of Stepan's pull request.

amb26 reviewed Oct 22, 2018

View reviewed changes

GPII-3138: Add uniqie ID to pouchManager temp test dir

dd3ea15

stepanstipl mentioned this pull request Oct 25, 2018

GPII-3138: Add unique ID to pouchManager temp test dir klown/universal#6

Merged

Merge pull request #6 from stepanstipl/dataloader-import3

8847cd3

GPII-3138: Add unique ID to pouchManager temp test dir

amb26 merged commit 8847cd3 into GPII:master Oct 25, 2018


		## Environment Variables

		- `COUCHDB_URL`: URL of the CouchDB database. (required)

GPII-3138: Move functionality of gpii-dataloader repo into universal #692

GPII-3138: Move functionality of gpii-dataloader repo into universal #692

Uh oh!

Conversation

klown commented Oct 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpii-bot commented Oct 18, 2018

Uh oh!

klown commented Oct 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stepanstipl commented Oct 22, 2018

Uh oh!

klown commented Oct 22, 2018

Uh oh!

klown commented Oct 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpii-bot commented Oct 22, 2018

Uh oh!

gpii-bot commented Oct 22, 2018

Uh oh!

gpii-bot commented Oct 22, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

klown Oct 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stepanstipl commented Oct 24, 2018

Uh oh!

gpii-bot commented Oct 24, 2018

Uh oh!

gpii-bot commented Oct 24, 2018

Uh oh!

klown commented Oct 24, 2018

Uh oh!

klown commented Oct 24, 2018

Uh oh!

gpii-bot commented Oct 24, 2018

Uh oh!

stepanstipl commented Oct 25, 2018

Uh oh!

amb26 commented Oct 25, 2018

Uh oh!

the-t-in-rtf commented Oct 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stepanstipl commented Oct 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpii-bot commented Oct 25, 2018

Uh oh!

stepanstipl commented Oct 25, 2018

Uh oh!

klown commented Oct 25, 2018

Uh oh!

stepanstipl commented Oct 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

klown commented Oct 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

the-t-in-rtf commented Oct 25, 2018

klown commented Oct 18, 2018 •

edited

Loading

klown commented Oct 18, 2018 •

edited

Loading

klown commented Oct 22, 2018 •

edited

Loading

klown Oct 23, 2018 •

edited

Loading

the-t-in-rtf commented Oct 25, 2018 •

edited

Loading

stepanstipl commented Oct 25, 2018 •

edited

Loading

stepanstipl commented Oct 25, 2018 •

edited

Loading

klown commented Oct 25, 2018 •

edited

Loading

the-t-in-rtf commented Oct 26, 2018 •

edited

Loading