Legacy and/or Deprecated Lookups

[paulhethmon]

Back in Confluence last month I asked:

I’m looking for a way to handle and express lookups that are no longer in use but exist in older listing data. As an example the field BusinessType might have a lookup of Video Store. 20 years ago that was important and used, today, not so much. I would like to expose the Video Store lookup in my metadata as a searchable item (albeit legacy) but I can’t allow it to be used on a new listing.
My search-fu this morning didn’t give me any leads on this, but it certainly seems like a situation many implementations would have.
Is there something in the standard I missed? Do we need to fill the gap?

[paulhethmon]

We discussed this briefly on the last Transport call. We don't have anything to serve this purpose today in the specifications.

Should we add this to the new Lookups Resource? If so, what permutations should there be:

• Deprecated
• Legacy
• Search Only
• Current

For me, the first 3 choices would evaluate to the same thing. You might see them in data and you can search, but you can't create a new listing with that value. Is one description better than the others? Are there other use cases?

[darnjo]

Hi Paul. Thanks for posting this.

We've been using the term "Legacy" in the language so far (for example, LegacyODataValue, and there could even be a LegacyRETSValue).

"Deprecated" might make sense if we want to differentiate these items from that terminology. Perhaps it's a YN field? DeprecatedYN (following other DD naming conventions)?

A couple thoughts:

• Would we want similar for the Field resource as well? https://ddwiki.reso.org/display/DDW17/Field+Resource
• It seems that searchability might be an attribute we'd want to capture separately? We had discussed having something to advertise "Capabilities" like Searchable or Updatable - which seems like it would apply to both fields and enumerations? Searchability is something that could apply to deprecated data elements or current ones.

[paulhethmon]

My concern about using YN fields is do they convey enough information. Does a value of Y in a LegacyYN field mean you can't use it at all? That it's still in the data? Or not still in the data? I can see various use cases (on both fields and lookups):

• Obsolete. Means it existed at one time but doesn't anymore.
• Legacy. Means it was used but is not current anymore. Existing data might have the value/field.
• SeachOnly. You can search for this field. You can't use it when adding a listing. (Maybe the same as legacy).
• Active/Current. Essentially the only value we have now.

Is something like Obsolete (with my definition) even needed?

[darnjo]

I think Legacy or Deprecated would be clearest.

Would you mind posting a sample payload showing how it might work with the Lookup Resource? https://ddwiki.reso.org/display/DDW17/Lookup+Resource

[SergioDelRioT4Bi]

I'm definitely in agreement with a new field in the Lookup Resource. I think it would be easier to keep it boolean value for implementation but am also not against making it a lookup field of it's own.

I feel this is also applicable to the Fields resource. Many vendors can't ever get rid of a field because nobody knows it will be removed. This would be now allow advertising of a field that is going away in 6 months, so take it out of your parsing, etc...

Hopefully the fields will be the same in each of Fields and Lookukp resource.

[bryanburgers]

We've been looking into this issue recently, too.

The problem we want to solve is that we anticipate deprecating fields and lookup in the future, and want to build in support now. So we're not dealing with "Oh, we deprecated that field 4 years ago before our RESO transition, now we want closed listings that use that lookup to still be valid after our transition" but rather "What will happen – even to active listings – when our schema changes in the future".

A solution we've been considering is using a timestamp instead of a boolean. Something like UnavailableAfterTimestamp. Listings with an OriginalEntryTimestamp that is before a field or lookup's UnavailableAfterTimestamp are allowed to use that field or lookup. Listings with an OriginalEntryTimestamp after UnavailableAfterTimestamp will get an error when trying to use that field or lookup.

It's not perfect, but it seems to be clean for an agent that wants to edit an existing active listing immediately after we remove support for a lookup.

[grispin]

Fields are often refactored rather then just being removed. It may be worth having a ReplacementFieldKey ReplacementLookupKey value that can inform the client where to find the replacement fields for the data.

Example cases:

• Add/Replace a Field/Lookup to be DD compliant
• Converting a fields from one data type to another (Number -> String, String -> Picklist, etc.)
• Restructuring a set of field to a different field structure (site parking, etc.)
• Spelling mistakes in key names
• Nomenclature updates (1/4th bath, Bay Window, etc.)

[paulhethmon]

I definitely think we should add this.

[paulhethmon]

A lot of good ideas in the discussion here. I do like the idea that @bryanburgers had about a date. We've recently had to introduce some rules where a field became required after a certain date, meaning that it wasn't required prior to that so the agents don't have to come up with additional information during an update to the listing. (Though a slightly different use case).

That follows along with managing field and lookup status.

For a Field, it seems we could have the following possibilities:

• Deprecated. A boolean indicating the current status
• FieldStatus. A lookup indicating the status. Now we could have deprecated, active, upcoming, going away soon
• DeprecatedDate. A date value of when a field became deprecated

In our system, we are only keeping the simple deprecated boolean so far. It feels like keeping something like UnavailableAfterDate is more of a rule than metadata about a field? There's a gray line there somewhere.

On the Lookup side, we've used that a bit more and keep a LookupStatus field that has 3 options:

• Active
• Deprecated
• Transition

Our internal definition of Transition is a lookup that might still exist in active listings but can no longer be selected. A Deprecated lookup could exist in historical listings.

[darnjo]

Personally, I like using versions for schema changes rather than dates/timestamps. But it might be hard to bring in a versioning scheme that would work for everyone here.

[bryanburgers]

We've recently had to introduce some rules where a field became required after a certain date, meaning that it wasn't required prior to that so the agents don't have to come up with additional information during an update to the listing.

Yeah, we’ve started doing this, too. Keying off something like OriginalEntryTimestamp so the requirements don’t change after a listing is created.

Action: SET_REQUIRED
Expression: OriginalEntryTimestamp > '2023-04-21T00:00:00Z'

[darnjo]

Approved by the Transport Workgroup in the Sept 2023 meeting as an RCP.

8 Yes, 0 No.

[darnjo]

See #97 for further discussion.