Tools to replicate sanitized real cases in dev/test environment #15976

yoomlam · 2021-03-08T22:44:50Z

Innovation Idea: Create a tool to replicate sanitized real cases in dev/test environment

Slack channel: #appeals-ip-replicate-data

Details: Many engineers desire a way to test solutions on real data without affecting prod data. The approach is to create a tool that can export an appeal and associated records, sanitize them of PII, then import them into a dev/test environment.

Hypothesis: If realistic appeals can be replicated in our dev/test environment, then we can test solutions locally and be more confident in the outcome of the solution. This can also be a building block to import more realistic cases into the default dev environment.

User Story: As a developer, I want to recreate sanitized instances of prod data in my dev (or testing) environment so that I can experiment with realistic data.

The data can be imported to be part of our seed data in the dev DB.
The data can be imported into the test DB for individual Rspec tests.

Description

Tools to replicate sanitized real cases in dev/test environment

Acceptance Criteria

Code compiles correctly
Able to export and import AMA appeals with tasks, issues, hearings, and associated users and orgs

Testing Plan

Copy files to prod (or ask Yoom to do it)

INSTANCE=i-06b8441957026bb62
scp lib/helpers/sanit*.rb  lib/helpers/association_wrapper.rb $INSTANCE:/tmp

ssm into the instance and move the new files to the Caseflow folder and start a Rails console in prod

sudo bash
cd /opt/caseflow-certification/src
mv -vf /tmp/sanit*.rb /tmp/association_wrapper.rb lib/helpers/

source /opt/caseflow-certification/caseflow-certification_env.sh
IRBRC=/tmp/my_repl.rb bin/rails c

In the Rails console, do the following

require "faker"
require "helpers/sanitized_json_configuration.rb"
require "helpers/sanitized_json_exporter.rb"

# Choose a random appeal
appeal=Appeal.find(rand(Appeal.count))
# Examine the appeal
appeal.treee
pp appeal.hearings
IntakeRenderer.patch_intake_classes
puts appeal.render_intake

# Export to a file
sje=SanitizedJsonExporter.new(appeal, verbosity: 5) # Maximize verbosity to see what it's doing
# scroll up to view the verbose output
# Save to file
sje.save('/tmp/appeal1.json', purpose: 'test1')

Note: Multiple appeals can be exported at one time: SanitizedJsonExporter.new(appeal1, appeal2)
4. Exit Rails console and examine the file for PII and sensitive data
5. Once satisfied that there is no sensitive data, copy the file locally. Back on your computer, copy the file:

scp $INSTANCE:'/tmp/appeal*.json' .

Then start a local Rails console and import the file.

sji = SanitizedJsonImporter.from_file('appeal1.json', verbosity: 5) # Maximize verbosity to see what it's doing
sji.import
# scroll up to view the verbose output

# Examine results
sji.imported_records # Hash of records imported into the database
sji.reused_records # Existing ActiveRecords that are not imported because they already exist in the database

# Compare appeal to the original in prod
IntakeRenderer.patch_intake_classes
sji.imported_records["appeals"].map{|appeal|
  appeal.treee
  pp appeal.hearings
  puts appeal.render_intake
}

Code Documentation Updates

Add or update code comments at the top of the class, module, and/or component.

va-bot · 2021-03-08T22:46:37Z

	1 Warning
⚠️	This is a Big PR. Try to break this down if possible. Stacked pull requests encourage more detailed and thorough code reviews

Generated by 🚫 Danger

codeclimate · 2021-03-08T22:49:59Z

Code Climate has analyzed commit c7af906 and detected 0 issues on this pull request.

View more on Code Climate.

yoomlam · 2021-03-18T17:00:00Z

lib/helpers/association_wrapper.rb

+  def typed_associations(excluding: nil)
+    belongs_to.having_type_field.except_fieldnames(excluding)
+  end
+
+  def typed_associations_with(assoc_class)
+    belongs_to.having_type_field.associated_with_type(assoc_class)
+  end
+
+  def untyped_associations_with(assoc_class)
+    belongs_to.without_type_field.associated_with_type(assoc_class)
+  end
+
+  def grouped_fieldnames_of_typed_associations_with(known_classes)
+    # Foreign keys that are not strings (e.g., Claimant.participant_id) involves
+    # more complex association that isn't currently handled (and may not need to be)
+    belongs_to.associations.group_by(&:class_name)
+      .slice(*known_classes)
+      .transform_values { |assocs| assocs.map(&:foreign_key).select { |fk| fk.is_a?(String) } }
+      .compact
+  end


These last 4 methods are used by SanitizedJsonConfiguration

yoomlam · 2021-03-18T17:02:35Z

lib/helpers/sanitized_json_configuration.rb

+# Configuration for exporting/importing data from/to Caseflow's database.
+# Needed by SanitizedJsonExporter and SanitizedJsonImporter.


This SanitizedJsonConfiguration is specific to the Caseflow domain, while SanitizedJsonExporter/Importer are meant to be domain-independent.

yoomlam · 2021-03-18T17:03:49Z

lib/helpers/sanitized_json_configuration.rb

+  module TaskAssignment
+    def assigned_to_user
+      select { |task| task.assigned_to_type == "User" }
+    end
+
+    def assigned_to_org
+      select { |task| task.assigned_to_type == "Organization" }
+    end
+
+    def with_type(task_type)
+      select { |task| task.type == task_type }
+    end
+  end


These methods are only used by the retrieval: lines above.

yoomlam · 2021-03-18T17:12:34Z

lib/helpers/sanitized_json_configuration.rb

+    end
+  end
+
+  USE_PROD_ORGANIZATION_IDS ||= false


This will be set to true later when we update Organization singletons in our seed data to have the same id as in prod to avoid problems with MailTeam.singleton returning the wrong record when a MailTeam org with a different id is imported.

lib/helpers/sanitized_json_configuration.rb

yoomlam · 2021-03-18T17:30:01Z

lib/helpers/sanitized_json_exporter.rb

+
+    return mapped_value if mapped_value
+
+    default_mapped_value(orig_value, field_name, **kwargs)


This only happens when a new field is added for sanitizing and none of the transforms return a non-nil.

lib/helpers/sanitized_json_importer.rb

spec/factories/cavc_remand.rb

yoomlam · 2021-03-19T15:08:02Z

Demo plan:

present slide
show export output in prod, explaining that CAVC remand appeal is associated with a source appeal

# CAVC remand with several issues
Appeal.where(stream_type: "court_remand", docket_type: "hearing").map{|a| [a.id, a.request_issues.count]}
=> [[133016, 4] ...]
appeal=Appeal.find(133016)
appeals=[appeal]
appeals.map &:treee
appeals.map{|a| puts a.render_intake}
sje=SanitizedJsonExporter.new(appeal, verbosity: 5);
sje.save('/tmp/appeal15.json', purpose: 'Mar 19th Demo')

scan through appeal15.json, noting obviously fake veteran_file_number/ssn/full_name/email and record ids are retained
describe rspec; run import within rspec, comparing task trees and intake trees (note similar record ids and css_ids for easy referencing with prod data (and consistency across many imports); timestamps are the same); note the source appeal for the CAVC remand is also imported

      sji = SanitizedJsonImporter.from_file('appeal15.json')
      imp_appeal1 = sji.import
      IntakeRenderer.patch_intake_classes  
      sji.imported_records[Appeal.table_name].map{|appeal|
        appeal.treee
        pp appeal.hearings
        puts appeal.render_intake
      }

describe SanitizedJsonConfiguration: @configuration
describe how SanitizationTransforms is applied to sanitize field values
SanitizedJsonExporter (~200 lines) and SanitizedJsonImporter (~300 lines) are domain-independent, which means they can be reused for other databases.

yoomlam · 2021-03-21T15:35:55Z

lib/helpers/sanitized_json_configuration.rb

+  def find_existing_record(clazz, obj_hash, importer: nil)
+    if clazz == User
+      # The index for css_id has an odd column name plus find_by_css_id is faster.
+      User.find_by_css_id(obj_hash["css_id"])


The User class has this curious index:

ActiveRecord::Base.connection.indexes(User.table_name).select(&:unique).first.columns => "upper((css_id)::text)"

instead of the typical Array:

ActiveRecord::Base.connection.indexes(Organization.table_name).select(&:unique).first.columns => ["url"]

lib/helpers/sanitized_json_importer.rb

yoomlam · 2021-03-21T15:38:10Z

lib/helpers/sanitized_json_configuration.rb

+      },
+      Veteran => {
+        track_imported_ids: true,
+        sanitize_fields: %w[file_number first_name last_name middle_name ssn],


I'd like to update the column comments in schema.rb so that these PII fields can be automatically identified, so we don't have to specify sanitize_fields manually here.

tomas-nava

Not quite done reviewing, but wanted to give you something to respond to.

I followed the testing plan and successfully exported & imported a relatively complex record. Most of my comments are minor. I will continue reviewing this week.

Something that occurred to me while using: you could put a check on SanitizedJsonImporter that stops it from importing if Rails.env.production?

lib/helpers/association_wrapper.rb

lib/helpers/sanitation_transforms.rb

spec/lib/helpers/sanitized_json_exporter_spec.rb

lib/helpers/sanitized_json_exporter.rb

lib/helpers/sanitized_json_importer.rb

Co-authored-by: Tomas Apodaca <Thomas.Apodaca@va.gov>

yoomlam · 2021-03-25T22:16:02Z

Added documentation: https://github.com/department-of-veterans-affairs/caseflow/wiki/Exporting-and-Importing-Appeals

tomas-nava

approve! with non-blocking questions/suggestions

lib/helpers/sanitized_json_importer.rb

lib/helpers/sanitized_json_exporter.rb

tomas-nava · 2021-03-24T00:09:10Z

lib/helpers/sanitized_json_exporter.rb

+    return unless obj_hash[field_name]
+
+    # Loop to ensure hash @value_mapping has a different value for each distinct key
+    10.times do


what happens if it's still not unique after 10 tries?

It uses the last generated value and moves on. The user can rerun the export as well.

### Description This PR doesn't affect end-users. It affects only the export utility in #15976 that engineers use. Improves #15976 by: * exports inactive `Organization` records so we don't have to manually create them in rspecs * eager loads Task associations * removes the now unnecessary `TaskAssignment` module * makes `AttorneyCaseReview` and `JudgeCaseReview` less specific to certain tasks * re-orders `AttorneyCaseReview` and `JudgeCaseReview` since they now depend on `Tasks` record to be retrieved ### Acceptance Criteria - [x] Tests pass ### Testing Plan None -- covered by RSpecs that use the export/import utility.

initial prototype

0cc207e

yoomlam self-assigned this Mar 8, 2021

yoomlam added 16 commits March 9, 2021 17:22

improve tests; better Organization handling

0390cd6

support multiple appeals

b32d4a4

refactor

9073279

refactor importer

d93e104

more refactoring

cfd0c92

fix console printout; add reek ignores

f232890

increase test coverage of exporter

d27a5c3

complete rspec for exporter

47b23ad

fix failed tests

79790ce

lint

d4baa68

Merge branch 'master' into yoom/export-import-from-prod

76258d3

bugfix: appeal ordering in JSON

7e9f61f

add difference method

b2207d8

lint

62cb97a

make SSN obviously fake

5e3dc23

add TaskTimer

146cd10

rubaiyat22 mentioned this pull request Mar 12, 2021

[Flaky Test] User switches hearing type to 'Virtual' #15997

Closed

6 tasks

yoomlam added 9 commits March 12, 2021 09:49

more code refactors

9299608

lint and fix test

5a86182

clean up

a90160c

add CavcRemand

9338051

extract SanitizedJsonDifference

34678c2

lint

edfab14

handle 'instructions' and array values

7432e90

Merge branch 'master' into yoom/export-import-from-prod

d44d304

export cavc_remand users too

f09068c

yoomlam added 3 commits March 18, 2021 09:00

Merge branch 'master' into yoom/export-import-from-prod

6e58ce2

add verbosity to exporter

f02930e

reek

f500ba3

yoomlam commented Mar 18, 2021

View reviewed changes

yoomlam added 2 commits March 18, 2021 12:48

clarify and clean up

8887ac7

allow configuration values to be modifiable

9b034fc

yoomlam added 7 commits March 19, 2021 11:11

add clarification comments

1c0f6fd

lint

b841b19

Merge branch 'master' into yoom/export-import-from-prod

9441187

generalize find_existing_record handling of OrganizationsUser

98f675e

more cleanup

13b49c6

rm adjust_identifiers_for_unique_records

4b29c6b

rm create_singleton

3fd4e61

yoomlam commented Mar 21, 2021

View reviewed changes

tomas-nava reviewed Mar 23, 2021

View reviewed changes

yoomlam and others added 3 commits March 23, 2021 08:03

Tomas's feedback

1fbd880

Co-authored-by: Tomas Apodaca <Thomas.Apodaca@va.gov>

syntactic changes based on feedback

24b0c4a

fix bugs around record reuse

be1e110

yoomlam mentioned this pull request Mar 23, 2021

Add export appeal endpoint for System Admins #16039

Closed

2 tasks

ignore RequestIssue.vacols_sequence_id for now

de9ade1

tomas-nava approved these changes Mar 25, 2021

View reviewed changes

yoomlam added 2 commits March 25, 2021 17:27

Merge branch 'master' into yoom/export-import-from-prod

a19bc4f

add precautions

c7af906

yoomlam added the Ready-to-Merge This PR is ready to be merged and will be picked up by va-bot to automatically merge to master label Mar 25, 2021

va-bot merged commit 8f971bc into master Mar 25, 2021

va-bot deleted the yoom/export-import-from-prod branch March 25, 2021 23:00

yoomlam mentioned this pull request May 14, 2021

Exporter: include inactive orgs in export #16250

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tools to replicate sanitized real cases in dev/test environment #15976

Tools to replicate sanitized real cases in dev/test environment #15976

yoomlam commented Mar 8, 2021 •

edited

va-bot commented Mar 8, 2021 •

edited

codeclimate bot commented Mar 8, 2021 •

edited

yoomlam Mar 18, 2021

yoomlam Mar 18, 2021

yoomlam Mar 18, 2021

yoomlam Mar 18, 2021

yoomlam Mar 18, 2021

yoomlam commented Mar 19, 2021 •

edited

yoomlam Mar 21, 2021

yoomlam Mar 21, 2021

tomas-nava left a comment •

edited

yoomlam commented Mar 25, 2021

tomas-nava left a comment

tomas-nava Mar 24, 2021

yoomlam Mar 25, 2021

		# Configuration for exporting/importing data from/to Caseflow's database.
		# Needed by SanitizedJsonExporter and SanitizedJsonImporter.


		return mapped_value if mapped_value

		default_mapped_value(orig_value, field_name, **kwargs)

Tools to replicate sanitized real cases in dev/test environment #15976

Tools to replicate sanitized real cases in dev/test environment #15976

Conversation

yoomlam commented Mar 8, 2021 • edited

Description

Acceptance Criteria

Testing Plan

Code Documentation Updates

va-bot commented Mar 8, 2021 • edited

codeclimate bot commented Mar 8, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yoomlam commented Mar 19, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomas-nava left a comment • edited

Choose a reason for hiding this comment

yoomlam commented Mar 25, 2021

tomas-nava left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yoomlam commented Mar 8, 2021 •

edited

va-bot commented Mar 8, 2021 •

edited

codeclimate bot commented Mar 8, 2021 •

edited

yoomlam commented Mar 19, 2021 •

edited

tomas-nava left a comment •

edited