Allow different ways to "update" field values

[mjordan]

Updates are complicated. Currently, updates in workbench "preserve any values in the fields, they don't replace the values." It would be useful to allow for this type of update, as well as updates that replace existing field values with ones in the input data, delete field values (#39), and remove a specific value from a multivalued field (#40).

It would also be useful to allow users to specify at the row level, rather than for all rows in the input CSV, which of these types of "update" is intended. One option (not necessarily the best) is to have the user provide a flag as the first character in the field itself that indicates which type of update they want. Typed relation fields already use a structured value using : to separate the parts of the value, so an approach like that might be applicable. For example, the user specifies an a for "append input values", an r for "replace all values with input", a d for "delete all existing values or a specific value" like this:

filed_my_field
a:additional value
d:
d:bad_value
r:new value

This would have to work with multivalued fields, and also with typed relation fields, which already use the :.

If the user didn't want row-level granularity for a given job (i.e., all update operations were of the same kind), the flag should be configurable in the config file.

[seth-shaw-unlv]

With an initial upload all content would be new.

However, with an update action I think the default assumption should be replace (that values I explicitly give the repository are the only ones that should persist). This option supports delete by simply removing the bad value from among the good that we hand back and add by simply adding the value to the list of values we are returning.

I think adding the column/row signifiers overly complicates this. Sure, we can provide this as an "added value" feature for those who want it, but I wouldn't require it by default.

[mjordan]

@seth-shaw-unlv nicely put. I'll continue with the "append" option just to get it working, but make "replace" the default.

[Natkeeran]

I've added a replace option here for multivalue taxonomy field.

[mjordan]

@Natkeeran sorry I've been distracted focusing on #171. Which branch did you use as the basis for the work in https://github.com/digitalutsc/islandora_workbench/commit/bae6e772a544a0007c84e95c8606d08bdcbd2129?

[Natkeeran]

@mjordan It is off the main branch. However, the logic is basic can can be adopted easily.

[mjordan]

As of today, users can now use a update_mode config option to indicate that update tasks "replace", "append", or "delete" field values for all supported Drupal field types, e.g. update_mode: append. The update mode applies to an entire update task, it isn't per-CSV row. "replace" is the default. Field handlers also now have over 4000 lines of test code.

New field types can be added by writing a class for them in "workbench_fields.py" and corresponding tests in "tests/field_tests.py".

Unless there are any objections, I'd like to close this issue. - @seth-shaw-unlv and @Natkeeran, OK to do so?

[mjordan]

Related issue #423.

[dara2]

Hi @mjordan - I just tested the update_mode: delete and although I am not getting any errors reported, the data in the field is not getting removed. Am I missing something? My config file looks like this:

task: update
host: "https://bd-i8-stage.born-digital.com/"
username: [my username]
password: [my password]
input_dir: i8demo_BD
input_csv: rights_delete.csv
validate_title_length: false
update_mode: delete

My CSV file is attached. Output from the .log file is:

06-Jun-22 11:17:41 - INFO - OK, connection to Drupal at https://bd-i8-stage.born-digital.com verified.
06-Jun-22 11:17:41 - INFO - OK, Islandora Workbench Integration module installed on https://bd-i8-stage.born-digital.com is at version 1.0.0.
06-Jun-22 11:17:41 - INFO - Client-side request caching is enabled.
06-Jun-22 11:17:42 - INFO - "Update" task started using config file i8demo_BD/update.yml.
06-Jun-22 11:18:18 - INFO - Node for https://bd-i8-stage.born-digital.com/node/1624 updated.
06-Jun-22 11:18:18 - INFO - Islandora Workbench successfully completed.

But the data is still in the Rights field:

rights_delete.csv

[mjordan]

Everything looks OK, but can you try again and put something in the field_rights CSV field? That shouldn't make a difference but I don't know what else to try without digging into the code. The update/delete task passes all its automated tests so there's something (for example, because it's blank it's being skipped) going on outside of the field-handling code to make this behave the way you are reporting.

[dara2]

So this is interesting: I ran it again, with text in the Rights field in my CSV, but I forgot to change the update_mode (so it was still set to "delete"). This time, the content was removed from the field completely (as I would have expected before).
Here's my revised config file (only change was to the CSV filename):

task: update
host: "https://bd-i8-stage.born-digital.com/"
username: [my username]
password: [my password]
input_dir: i8demo_BD
input_csv: rights_replace.csv
validate_title_length: false
update_mode: delete

My revised CSV (now with content in the Rights column) is attached. You can see the field data is now gone. So it seems to obey the "delete" configuration...if there IS content in the Rights column?

rights_replace.csv

[mjordan]

Without any hands-on verification/testing, I suspect that the fact that the field was blank is causing it to be skipped by the field handler, where the nulling-out of the field contents happens before Workbench updates Drupal. I'll test this hypothesis this evening and fix if that's it.

[dara2]

Okay, thanks! Just to clarify: isn't that how the "delete" mode is meant to work - if you run an update with blank columns, it deletes what is in that field for the nodes you indicate? Or am I misunderstanding its purpose?

[mjordan]

You're not misunderstanding its purpose - you found a bug.

[dara2]

Haha - okay, just wanted to make sure! 😄

[mjordan]

OK, I've fixed this. Thanks for catching it.

[mjordan]

Closing since we now have 3 different ways to update nodes!