Separate Part Name from Part Preservation

[campmlc]

Proposal to separate part name (e.g. "tissue") from part preservation (e.g. "frozen").
Posting this as new issue related to repeated earlier discussion.

From email string:
These are clearly part attributes - they apply to parts, there can be many of them, they're determinations, they're time-dependent, etc. You can add them with the "add more" widget on data entry. We'll need to set up code table values before you can use them - that's fairly trivial, you can do it under edit code tables.

They can be bulkloaded with the parts bulkloader, and are in partdetail in specimen results/downloads.

I agree that preservation should not be confounded with part name. Getting it out affects lots of things (labels, the "tissues" search, data entry, etc.) so will need some discussion. Issue please.

---

Current status: https://docs.google.com/spreadsheets/d/1AY1EXQzKUSAg9EvZFjayTXcCBciXd_syCxt0GSX2tww/edit#gid=485619714 will serve as a migration path on 2021-07-01

[campmlc]

Previous issues: #212, #319, #352, #991, #1020, #1084, #1119, #1131, #1203, #1384

[dustymc]

Thanks @campmlc !

This isn't (yet...) a technical issue, so I'm unassigning me. The structure to do this all exists, I think we just have to work out how to use it (and/or decide if we can use it).

I think a model in which attributes of a physical object are in a 1-->many relationship with that object is "correct" (more correct, anyway) from a data modeling standpoint, but I'm also not entirely sure it's usable. Given the choice between correct and usable, I'll always (try to!) take usable.

#212, #991, #1020: maybe we should toss the whole part model and start blank-slate - the current model has known problems with no clear solutions.

Also for any developing wish-list: find all the ribs. They're currently under "rib" and "body" and "whole organism" and "embryo" and (sorta/maybe) "fossil" and ....

#319 got revived for and implemented as #800. It's clearly "more correct" as preservation history than what we're proposing here (eg, "frozen" is ambiguous, TEMPERATURE at TIME isn't) but I think we decided it wasn't usable as part of the GGBN work.

#352 introduced the structure which allowed this.

#1084: #1084 (comment) is the summary (we can't articulate what a "tissue" is).

#1119 is pretty much this and #319.

#1131 and #1203 are a bit like #1084 (comment) - we did some stuff, things definitely got better for it, but "is this a part?" still has a very arbitrary answer.

#1384 just looks like another symptom of #319 to me - qualitative labels (excellent, rotten) for what is essentially quantitative data (time at temperature) are weird (albeit perhaps necessary if anyone's going to interact with the data!).

[dustymc]

find/make example in test, post here

[Jegelewicz]

AWG agrees separation would be better. Use a part attribute to maintain history. Add part attributes to data entry and bulkloader. Controlled vocabulary unless someone has a good reason not to. @dustymc ? put an example here for us to see how it will look.

[dustymc]

http://arctos-test.tacc.utexas.edu/guid/CHAS:Bird:17187

[Jegelewicz]

This doesn't look any different from what we have now.

I think we were expecting
part=DNA
preparation=frozen

[campmlc]

What this issue refers to is splitting "DNA" from "frozen", or any physical part from it's fixation or preservation medium. This means everything in parentheses needs to be transferred to one or more controlled vocabulary fields. In many cases, we will need more than one ("liver (95% ethanol, frozen)" = part name "liver"; fixation "95% ethanol", storage: "frozen". If we use part attributes for this, I'd like them to be in an expandable dropdown so that we don't have to look at the entire list for each part unless we click "expand".

…

On Wed, Aug 1, 2018 at 1:33 PM, Teresa Mayfield ***@***.***> wrote: This doesn't look any different from what we have now. [image: image] <https://user-images.githubusercontent.com/5725767/43544115-33fe1b4e-958f-11e8-9585-543e74540c20.png> I think we were expecting part=DNA preparation=frozen — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/ArctosDB/arctos/issues/1460#issuecomment-409694849>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOH0hHA_ZUeQBJBXB7ng9WSbeIuIIt0Rks5uMgKSgaJpZM4SXYNg> .

[dustymc]

doesn't look any different

It's not, which is why I've been saying "go for it!" for some time now.

part=DNA
preparation=frozen

That's just data and doesn't change any functionality. I'm happy to help with updates and such, if you want to do something in your collection or can convince other collections to come along or whatever.

expandable dropdown so that we don't have to look at the entire list for each part unless we click "expand".

I'm not sure what this means.

[campmlc]

Great. I believe we have consensus to make this a global change. So we would need you to split out everything in parentheses, and add a separate part preparation field that allows multiple concatenated searchable values into the parts view. This would need to be on same line as disposition, condition, etc. as follows: Part Name: "liver" Part Preparation: "95% ethanol; frozen" Part Disposition: "in collection" Part Condition: "good" The last comment refers to the part attributes display. I am requesting to make this an expandable drop-down visible only when clicked, to reduce the overwhelming visual clutter on the page. On Aug 1, 2018 2:00 PM, "dustymc" <notifications@github.com> wrote: doesn't look any different It's not, which is why I've been saying "go for it!" for some time now. part=DNA preparation=frozen That's just data and doesn't change any functionality. I'm happy to help with updates and such, if you want to do something in your collection or can convince other collections to come along or whatever. expandable dropdown so that we don't have to look at the entire list for each part unless we click "expand". I'm not sure what this means. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/ArctosDB/arctos/issues/1460#issuecomment-409702539>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOH0hE1zZNCQ_DX0tCTAGF4bnOaG_TGrks5uMgjJgaJpZM4SXYNg> .

[dustymc]

consensus

I hope so, but this should definitely go to the group.

add a separate part preparation field

That is a part attribute; there's nothing to add.

multiple concatenated searchable values

Hu? http://arctos-test.tacc.utexas.edu/guid/CHAS:Bird:17187 has change over time.

You'd concatenate it into "postcranial skeleton (rotten, spiffy, rotten, spiffy)"??

part attributes display...visible only when clicked

I think that's a huge problem. The thing people actually look at just says "liver" so they request tissue samples or etc - the DATA are liver (in formalin, which you can't see). That seems like it's going to cause usability problems.

[campmlc]

The preservation is just as important as condition and disposition, so it should display on the main line of the part field. I don't care if is an attribute in the underlying data table. The data structure does not have to determine the visual display. I'm suggesting, as others have previously done, that having all the part attributes visible for all parts makes the visual display virtually impossible. So, add preservation as one of the cells in the top row, and if there is more than one value, concatentate them, as we do with the json string in part location path. Keep it in the attribute table too, if necessary, with the dates etc. But make the part attribute table optionally visible with an expand function. We could actually do the same with condition -and just show the most recent attribute value . . . Happy to discuss this with the AWG, but we already have had multiple people complain about the messiness of the current attribute display.

…

On Wed, Aug 1, 2018 at 3:54 PM, dustymc ***@***.***> wrote: consensus I hope so, but this should definitely go to the group. add a separate part preparation field That is a part attribute; there's nothing to add. multiple concatenated searchable values Hu? http://arctos-test.tacc.utexas.edu/guid/CHAS:Bird:17187 has change over time. [image: screen shot 2018-08-01 at 2 53 56 pm] <https://user-images.githubusercontent.com/5720791/43551126-c2a679c6-959a-11e8-87b6-e1ec184f6e1f.png> You'd concatenate it into "postcranial skeleton (rotten, spiffy, rotten, spiffy)"?? part attributes display...visible only when clicked I think that's a huge problem. The thing people actually look at just says "liver" so they request tissue samples or etc - the DATA are liver (in formalin, which you can't see). That seems like it's going to cause usability problems. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/ArctosDB/arctos/issues/1460#issuecomment-409737178>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOH0hMKA_dSDjpS8MnUIqtfmsiTA-PB_ks5uMiOpgaJpZM4SXYNg> .

[dustymc]

preservation is just as important as condition and disposition

Moreso maybe, but it's a different type of data.

concatentate them, as we do with the json string in part location path

Those are different things - JSON is capable of carrying the "this is not the current determination" data and would satisfy my concerns.

In any case I think that's details - we're all merged on storing the data as part attributes, correct?

[acdoll]

I think we are asking to separate 'preservation' out as a different type of data so that we can have something like this:

That will allow multiple concurrent preservation types (and also could be tracked over time).

[campmlc]

Yes, that is my understanding. Thanks for the demo, Andy.

…

On Wed, Aug 1, 2018 at 6:49 PM, Andrew Doll ***@***.***> wrote: I think we are asking to separate 'preservation' out as a different type of data so that we can have something like this: [image: image] <https://user-images.githubusercontent.com/12507297/43556164-8e6b63a8-95bb-11e8-8dc5-f7187eb79083.png> That will allow multiple concurrent preservation types (and also could be tracked over time). — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/ArctosDB/arctos/issues/1460#issuecomment-409770092>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOH0hM06A2TYhbMYv1uLq1ln5MGOtKcMks5uMkyhgaJpZM4SXYNg> .

[dustymc]

AWG:

• we DO want to do this for all Arctos
• minimize default info - add expand button - remember last action??
• default part name (latest attribute inf) [clicky2expand]

[campmlc]

Separate part from preservation; use part attributes, but show preservation as most recent preservation in part attributes.
All/none expand/collapse of part attributes. Default is that they are not visible.
AWG 8-9-18

[KyndallH]

Where are we with this??

[dustymc]

If we're forcing this on everyone, we need to let them know and I need a lookup table to make changes.

If you just want to use it, go ahead - this changes nothing in the model and has always been possible.

[Jegelewicz]

this changes nothing in the model and has always been possible.

I disagree. Parts are currently named "part (preservation)". This would be a pretty big change and it is one we have been asking for since I think June or July. I don't think any of us believe we have seen a demonstration of the requested change in action and perhaps this needs a committee to get it done.

[dustymc]

That is not a model change, it's just data (and perhaps accompanying authorities).

This may end up requiring a new "part preservation" (and maybe other stuff) code table(s) under http://arctos.database.museum/info/ctDocumentation.cfm?table=CTSPECPART_ATTRIBUTE_TYPE.

This may lead to form changes, but as always I very strongly prefer to build that around existing data - I'll tackle that after someone's used this.

If this involves controlled vocabulary, then it will work exactly the same was as "tissue quality." Tissue quality uses http://arctos.database.museum/info/ctDocumentation.cfm?table=CTTISSUE_QUALITY because http://arctos.database.museum/info/ctDocumentation.cfm?table=CTSPEC_PART_ATT_ATT.

If it involves free text, it will work exactly the same as "part location."

http://arctos-test.tacc.utexas.edu/guid/CHAS:Bird:17187 contains many controlled and uncontrolled part attributes, and should adequately demonstrate any possible arrangement that this could take.

[KyndallH]

It would be controlled text.

Pretty much what currently is in () in the part code table would become preservation. So blood (flash-frozen) would be blood with flash-frozen in preservation. Blood would simply be blood.

[dustymc]

preservation

There's an eternal argument about what's preservation and what's storage and etc. in there, but I'm pretty happy to ignore that for now if ya'll are.

currently is in ()

create table temp_pn_has_parens as select distinct part_name,
regexp_substr(part_name, '\(([^\)]+)\)', 1,1,NULL,1) stuff_from_parens
from ctspecimen_part_name where part_name like '%(%';

temp_pn_has_parens.csv.zip

create table temp_pn_parendata as select distinct stuff_from_parens from temp_pn_has_parens;

temp_pn_parendata.csv.zip

If you want to add and populate a description column to that I can create a code table.

Some of these should (probably?) be multiple attributes - "formalin-fixed, 55% isopropanol," "70% ethanol, frozen," etc.

[campmlc]

We want to get away from the multiple attributes, as that causes the exponential expansion of the code table. I had thought we'd use part attributes to track history and multiple fixation/preservation categories, and only have the most recent in the "preservation" display field. We are trying to simplify something with a complex history. Or, can we concatenate all the terms, so they can be chosen separately at data entry and added to over time in part attributes, but display as a concatenated preservation field when more than one term exists?

…

On Tue, Oct 30, 2018 at 4:44 PM dustymc ***@***.***> wrote: preservation There's an eternal argument about what's preservation and what's storage and etc. in there, but I'm pretty happy to ignore that for now if ya'll are. currently is in () create table temp_pn_has_parens as select distinct part_name, regexp_substr(part_name, '\(([^\)]+)\)', 1,1,NULL,1) stuff_from_parens from ctspecimen_part_name where part_name like '%(%'; temp_pn_has_parens.csv.zip <https://github.com/ArctosDB/arctos/files/2531928/temp_pn_has_parens.csv.zip> create table temp_pn_parendata as select distinct stuff_from_parens from temp_pn_has_parens; temp_pn_parendata.csv.zip <https://github.com/ArctosDB/arctos/files/2531931/temp_pn_parendata.csv.zip> If you want to add and populate a description column to that I can create a code table. Some of these should (probably?) be multiple attributes - "formalin-fixed, 55% isopropanol," "70% ethanol, frozen," etc. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/ArctosDB/arctos/issues/1460#issuecomment-434496947>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOH0hEAynCLvSiVaPypSjq6q_IAS-Wzfks5uqNZWgaJpZM4SXYNg> .

[Jegelewicz]

@Nicole-Ridgwell-NMMNHS thanks! changes made and I gave you code table access.

[Nicole-Ridgwell-NMMNHS]

Thanks!

[ebraker]

Looks great Teresa, thanks for doing this! There are a few times where we have a base definition, but where preservation method is included in the definition for one of the variants. Do you want me to point these out, or these will get taken care of (merged with base definition, with preservation defined separately)?
e.g.,

eye: An organ of the visual system. Eyes provide animals with vision, the ability to receive and process visual detail, as well as enabling several photo response functions that are independent of vision. Eyes detect light and convert it into electro-chemical impulses in neurons. Wikipedia
vs.
eye (formalin): eye fixed/preserved in formalin

egg: The organic vessel containing the zygote in which an embryo develops until it can survive on its own, at which point the animal hatches. Wikipedia
vs.
egg (70% ethanol): whole ovum, stored in a solution of 50% isopropanol [[[this one also doesn't make sense]]]

[dustymc]

@ebraker no parentheses-having part should survive this process.

I updated the original issue with the migration path spreadsheet, so I don't have to go dig it out of github's horrid little collapsy-thing.

[dustymc]

migration path is now clean

[Jegelewicz]

@dustymc when I Google "hindquarters" this is what I get...

https://en.wikipedia.org/wiki/Buttocks#Synonyms

Enjoy

[Jegelewicz]

@dustymc these are the compound parts that you can add.
compound_part_adds.zip

[dustymc]

thx/done

[ccicero]

@dustymc did the 'clean-up' of MVZ parts that also have a separate preservation method get resolved - so we don't end up with a dup preservation for those records? If there are still MVZ records that have preservation in both the part name and preservation field, can you send me a CSV file of those? Thanks!

[campmlc]

Coming into this late after being out of town - what is needed for the following, and why are they "not parts"? Not parts: part ------------------------------ tissue block specimen subsample stomach plus crop stomach plus stomach content tongue, trachea, and syrinx nasal flush skin, shed specimen subsample stomach plus crop content stomach plus stomach content tail tip serum skin, shed stomach plus crop content tail tip wing, flat wing, spread carcass, dissected carcass, headless carcass, skinned tail tip blood serum, muscle brain, eye culture egg contents heart, kidney, liver, lung heart, kidney, liver, muscle heart, kidney, muscle heart, muscle kidney, liver kidney, liver, spleen kidney, lung kidney, lung, muscle kidney, muscle kidney, spleen liver, lung liver, muscle lung, muscle muscle, eye serum specimen subsample swab, nasal swab, oral and cloacal tail tip tissue extract hindquarters spleen, lung, kidney, fat swab, cloacal swab, oral and cloacal specimen in sections strobila, partial testis, sectioned

…

On Mon, Jun 21, 2021 at 11:25 AM dustymc ***@***.***> wrote: * [EXTERNAL]* Not parts: part ------------------------------ tissue block specimen subsample stomach plus crop stomach plus stomach content tongue, trachea, and syrinx nasal flush skin, shed specimen subsample stomach plus crop content stomach plus stomach content tail tip serum skin, shed stomach plus crop content tail tip wing, flat wing, spread carcass, dissected carcass, headless carcass, skinned tail tip blood serum, muscle brain, eye culture egg contents heart, kidney, liver, lung heart, kidney, liver, muscle heart, kidney, muscle heart, muscle kidney, liver kidney, liver, spleen kidney, lung kidney, lung, muscle kidney, muscle kidney, spleen liver, lung liver, muscle lung, muscle muscle, eye serum specimen subsample swab, nasal swab, oral and cloacal tail tip tissue extract hindquarters spleen, lung, kidney, fat swab, cloacal swab, oral and cloacal specimen in sections strobila, partial testis, sectioned — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/ArctosDB/arctos/issues/1460#issuecomment-865213727>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADQ7JBCVHGZK2BHI7FZNCPLTT5YZBANCNFSM4ES5QNQA> .

[Jegelewicz]

Pretty sure those are all parts...

[dustymc]

@ccicero I did now, thanks for the reminder.

Do we need a reminder that this is happening Thursday?

drop table temp_dpp;

create table temp_dpp as select 
  guid,
  specimen_part.collection_object_id pid,
  part_name,
  attribute_type,
  attribute_value,
  attribute_units,
  determined_by_agent_id,
  attribute_remark,
  determined_date
from flat 
inner join specimen_part on flat.collection_object_id=specimen_part.derived_from_cat_item 
inner join specimen_part_attribute on specimen_part.collection_object_id=specimen_part_attribute.collection_object_id 
where part_name like '%(%' and attribute_type='preservation';

delete from specimen_part_attribute where
  attribute_type='preservation' and collection_object_id in (
    select pid from temp_dpp where guid like 'MVZ:Mamm:%'
);

delete from specimen_part_attribute where
  attribute_type='preservation' and collection_object_id in (
    select pid from temp_dpp where guid like 'MVZ:Bird:%'
);

[Jegelewicz]

I just changed the banner - @ewommack can you make an annoucement to the AWG and Arctos Goolge group?

[dustymc]

Part names have been updated and attributes added in test.

[ccicero]

thanks @dustymc

[Jegelewicz]

OOOOOOOH! The little "summary" thingee is cool!!!!

[Jegelewicz]

How long before this updates? I'm sure the questions will come up...

[dustymc]

bulkload parts and select specific events?

Maybe another reason to catalog "Occurrences."

How long before this updates?

In test - a bit after I get https://github.com/ArctosDB/internal/issues/65. In production, it'll probably take a few days - Monday, hopefully, assuming all the "interesting" happens today and not tomorrow.

I've been running the update manually, but it's safe to assume it's never going to fully complete. I got that one.

"summary" thingee

Well the mechanism behind it might turn out to be neat, anyway - should be possible to swap in about any flavor/number of parts views on demand now.

[bryansmclean]

are bulkload operations paused in any way today?

[dustymc]

Yes bulkloading and changes becoming visible in the UI are likely to be very slow to nonexistent.

[Jegelewicz]

@dustymc can we close this (well send to needs documentation)?

[Jegelewicz]

Transferring back from documentation so we can find it.