Skip to content

Commit

Permalink
Remove remediation-related code, docs and tests
Browse files Browse the repository at this point in the history 
Fixes #278
  • Loading branch information
@atz
atz committed Sep 17, 2018
1 parent 3069ea3 commit f337865
Show file tree
Hide file tree
Showing 5 changed files with 0 additions and 414 deletions.
32 changes: 0 additions & 32 deletions .rubocop_todo.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -101,11 +101,6 @@ Lint/DuplicateMethods:
Exclude:
- 'app/lib/pre_assembly/object_file.rb'

# Offense count: 1
Lint/EnsureReturn:
Exclude:
- 'app/lib/pre_assembly/remediation/item.rb'

# Offense count: 1
Lint/ParenthesesAsGroupedExpression:
Exclude:
Expand All @@ -115,7 +110,6 @@ Lint/ParenthesesAsGroupedExpression:
Lint/RescueException:
Exclude:
- 'app/lib/pre_assembly/bundle.rb'
- 'app/lib/pre_assembly/remediation/item.rb'
- 'app/lib/pre_assembly/reporting.rb'

# Offense count: 2
Expand Down Expand Up @@ -186,11 +180,6 @@ Metrics/ModuleLength:
Metrics/PerceivedComplexity:
Max: 73

# Offense count: 1
Naming/AccessorMethodName:
Exclude:
- 'app/lib/pre_assembly/remediation/item.rb'

# Offense count: 7
# Configuration parameters: Blacklist.
# Blacklist: (?-mix:(^|\s)(EO[A-Z]{1}|END)(\s|$))
Expand All @@ -199,13 +188,6 @@ Naming/HeredocDelimiterNaming:
- 'spec/lib/pre_assembly/digital_object_spec.rb'
- 'spec/lib/pre_assembly/smpl_spec.rb'

# Offense count: 1
# Configuration parameters: EnforcedStyle.
# SupportedStyles: snake_case, camelCase
Naming/MethodName:
Exclude:
- 'app/lib/pre_assembly/remediation/item.rb'

# Offense count: 2
# Configuration parameters: NamePrefix, NamePrefixBlacklist, NameWhitelist, MethodDefinitionMacros.
# NamePrefix: is_, has_, have_
Expand All @@ -215,7 +197,6 @@ Naming/MethodName:
Naming/PredicateName:
Exclude:
- 'spec/**/*'
- 'app/lib/pre_assembly/remediation/item.rb'

# Offense count: 2
# Configuration parameters: MinNameLength, AllowNamesEndingInNumbers, AllowedNames, ForbiddenNames.
Expand Down Expand Up @@ -355,7 +336,6 @@ Security/YAMLLoad:
# IgnoredMethods: lambda, proc, it
Style/BlockDelimiters:
Exclude:
- 'app/lib/pre_assembly/remediation/item.rb'
- 'app/lib/pre_assembly/smpl.rb'
- 'spec/jobs/discovery_report_job_spec.rb'
- 'spec/jobs/preassembly_spec.rb'
Expand All @@ -367,7 +347,6 @@ Style/BlockDelimiters:
# SupportedStyles: braces, no_braces, context_dependent
Style/BracesAroundHashParameters:
Exclude:
- 'app/lib/pre_assembly/remediation/item.rb'
- 'config/deploy.rb'
- 'spec/lib/pre_assembly/bundle_spec.rb'

Expand Down Expand Up @@ -452,7 +431,6 @@ Style/HashSyntax:
- 'app/lib/pre_assembly/bundle.rb'
- 'app/lib/pre_assembly/digital_object.rb'
- 'app/lib/pre_assembly/logging.rb'
- 'app/lib/pre_assembly/remediation/item.rb'
- 'app/lib/pre_assembly/reporting.rb'
- 'app/lib/pre_assembly/reporting_old.rb'
- 'app/lib/pre_assembly/smpl.rb'
Expand Down Expand Up @@ -494,7 +472,6 @@ Style/MutableConstant:
- 'app/lib/bundle_context_temporary.rb'
- 'app/lib/pre_assembly/digital_object.rb'
- 'app/lib/pre_assembly/logging.rb'
- 'app/lib/pre_assembly/remediation/item.rb'

# Offense count: 3
# Cop supports --auto-correct.
Expand Down Expand Up @@ -558,12 +535,6 @@ Style/RedundantException:
Exclude:
- 'app/lib/pre_assembly/digital_object.rb'

# Offense count: 2
# Cop supports --auto-correct.
Style/RedundantParentheses:
Exclude:
- 'app/lib/pre_assembly/remediation/item.rb'

# Offense count: 2
# Cop supports --auto-correct.
# Configuration parameters: AllowMultipleReturnValues.
Expand All @@ -577,7 +548,6 @@ Style/RedundantReturn:
# SupportedStyles: implicit, explicit
Style/RescueStandardError:
Exclude:
- 'app/lib/pre_assembly/remediation/item.rb'
- 'app/lib/pre_assembly/reporting_old.rb'

# Offense count: 320
Expand Down Expand Up @@ -612,7 +582,6 @@ Style/SymbolProc:
# SupportedStyles: require_parentheses, require_no_parentheses, require_parentheses_when_complex
Style/TernaryParentheses:
Exclude:
- 'app/lib/pre_assembly/remediation/item.rb'
- 'app/lib/pre_assembly/reporting.rb'
- 'app/lib/pre_assembly/reporting_old.rb'
- 'app/lib/pre_assembly/smpl.rb'
Expand Down Expand Up @@ -649,7 +618,6 @@ Style/TrivialAccessors:
# Cop supports --auto-correct.
Style/UnneededInterpolation:
Exclude:
- 'app/lib/pre_assembly/remediation/item.rb'
- 'spec/lib/pre_assembly/digital_object_spec.rb'

# Offense count: 1
Expand Down
136 changes: 0 additions & 136 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -820,139 +820,3 @@ steps={'assemblyWF' => ['checksum-compute'],'accessionWF' => ['content-metadata
Assembly::Utils.reset_workflow_states(:druids=>druids,:steps=>steps)
```

## Remediation

### Basic Object Remediation

A basic object remediation framework is provided for mass remediating objects.

To use it, first decide how you will specify the list of druids. You can
either manually specify with a CSV (see step 1 below), or use the pre-assembly
YAML log file if you are remediated a run from pre-assembly.

1. For running a specific list of druids, create a CSV file containing a list
of druids you wish to remediate, one line per druid, with a header column
called "druid" or "Druid" (caps don't matter).

There can be other columns too -- they will be ignored. They can be either
full druids (with the prefix) or just the PIDs. Save it somewhere the
pre-assembly code can read it.

e.g. `druid druid:oo000oo0001 druid:oo000oo0002 oo000oo0003`

1. Create a ruby file that defines exactly how you need to remediate each
object. The ruby file will define a method that has access to the

fedora object and can perform any actions needed on the object. The actual
loading, saving, and versioning of objects is handled by the framework - you
just need to define the actual logic needed to operate on the object. The
file needs to have a specific format with two specific methods that must be
defined, one that determines if remediation will occur at all, and the second
indicates the type of remediation to perform. An example file and its format
is shown in the file 'lib/remediation/remediate_project_example.rb' Don't edit
that file - copy it, and edit it somewhere the script can read it.

Note that you have access to [`equivalent-xml`](https://github.com/mbklein/equivalent-xml)
for comparing xml when deciding if you need to remediate. You also can use nokogiri.
If you use `equivalent-xml` be sure to require it at the top of your script.

1. Run with the command below (presumably on the `lyberservices-prod` server,
which has access to Fedora production, although you can also run in test or
development mode if you can access those environments). Pass
in the either the CSV file with DRUIDs or the pre-assembly YAML file and the
Ruby file you generated in steps 1 and 2. You will get two output files --
the first is a CSV file which includes columns indicating if remediation
succeeded, a message and a timestamp. You will also get a .log file. Each
file is named in the same way as the input file with `_log.csv` and `_log.yml`
appended to the filename and is placed in the same location. This means you
should place the input CSV in be a location where the script will have write
access to that location (i.e. not on a thumper drive if you are on
`lyberservices-prod`).

Note that the `_log.yml` file is used to ensure that objects are not run through
remediation twice, so you should keep that file in the same location as the
input CSV if you need to resume a large remediation. In each case, the file
will always be appended to.
```bash
RAILS_ENV=production bin/remediate INPUT_FILE.CSV[or LOG_FILE.YAML]
REMEDIATE_LOGIC_FILENAME.rb
```
For long running actions, you can run in nohup mode:
```bash
RAILS_ENV=production nohup bin/remediate INPUT_FILE.CSV[or
LOG_FILE.YAML] REMEDIATE_LOGIC_FILENAME.rb &
```

The result will be some screen output and a detailed log file in .YML format
in the same location and with the same name as the input CSV file. The input
CSV file will also be updated with two additional columns - a status
indicating if remediation succeeded and a message. You can re-run the
remediation with the same CSV file and it will automatically skip already
completed objects - SO KEEP THE CSV FILE.

You can also add optional parameters to override defaults for checking if
items are in accessioning or if they require versioning and set to false. Only
do this if you are sure you know what you are doing for scripts that for some
reason will not work with versioning!
```bash
RAILS_ENV=production bin/remediate INPUT_FILE.csv
REMEDIATE_LOGIC_FILENAME.rb [PAUSE_TIME_IN_SECONDS] ignore_versioning
ignore_accessioning
```

Finally, you can specify a pause time in seconds per object. This can be
useful for large remediation jobs that are a low priority, which allows them
to run at a slower pace and thus generating less concurrent load on the system
while other higher priority jobs are running concurrently.

### MODs Remediation

A custom script has already been written to handle the case of updating MODs
given a spreadsheet and a MODs template. See the file
`devel/update_mods_metadata.rb` for more information.

### Custom Object Remediation

You can also build a fully customized remediation script that does not require
input DRUIDs in a CSV and can pass data to individual druids for more specific
remediation. To do this, you will need to implement the logging or resume
capability you will need. You will also still implement your own project
specific Ruby file (as in step 2 above) but you will also implement your own
mechanism for generating druids to remediate and for logging their completion.

Within the pre-assembly codebase, you will have access to the following
remediation class. Pass it a druid and your remediation logic Ruby file, and
it will return you success or failure and a message. You can also use the
built in logging methods to record success/failure to help with resuming. You
can also optionally pass any data structure or object you need and have that
available in your custom method.

```ruby
project_file = 'my_remediation.rb'
pid = 'druid:oo000oo0001'
require project_file

item = PreAssembly::Remediation::Item.new(pid, optionalDataStructure) # optionalDataStructure is some kind of object, string, hash, etc.
# it will be made available to your "remediate_logic" method in
# the instance variable @data
item.extend(RemediationLogic) # add in our project specific methods
item.description = 'some description that will be added to any versions which are opened' # optional
success = item.remediate # returns true or false
puts item.message # some more info about what was done
```

Loop over your PIDs and log results as needed. You can log to an output CSV
and/or a YML file if you wish, using the following methods
```ruby
item.log_to_progress_file(progress_log_file) # pass in a fully qualified path to a YML file to append to
item.log_to_csv(csv_out) # pass in a fully qualified path to a CSV file to append to
```

You can read in completed druids from a progress_log_file you have previously
created using a class level method if you wish to check if a druid is already
completed. Each of these calls gives you an array of druids.
```ruby
completed_druids = PreAssembly::Remediation::Item.get_druids(progress_log_file)
failed_druids = PreAssembly::Remediation::Item.get_druids(progress_log_file, false)
done if completed_druids.include?(pid) # will give you a true or false if your current pid is already done
```

0 comments on commit f337865

Please sign in to comment.