feat(search): Add search for field level description and tags #2491
Conversation
| /** | ||
| * List of field descriptions | ||
| */ | ||
| editedFieldDescriptions: optional array[string] |
There was a problem hiding this comment.
My understanding is that fieldDescriptions is coming from SchemaMetadata, whereas editedFieldDescriptions is coming from EditableSchemaMetadata. But what is the source of truth when it comes to field description? If a user has made changes to a field description via UI, shouldn't that be the "updated" field description? Why do we need to keep both?
Not sure if this has been discussed before. If yes, please feel free to point to relevant doc/rfc.
There was a problem hiding this comment.
That's a good question @jywadhwani
I'm inclined to let this implementation stay for the following reasons:
- We should keep the "human written" documentation separate from the "schema-attached" documentation (we are already doing that in the db, so makes sense to continue doing it in the indexes). This way, we don't mix the two streams of information (especially without a governed approval flow for documentation overrides). In fact, we have been asked to surface both the pieces of information in the UI from the community (this is not present in the app currently).
- Once we build that sort of approval flow in the application, we can store breadcrumbs for when an editable schema field description overrides an existing schema field description (imagine a flag on the schema-field-description marking it as not-to-be-used), which would trigger the indexer to remove this text from the index.
Hope this addresses your concern.
There was a problem hiding this comment.
Sounds good, thanks for the explanation @shirshanka.
Add search for field level description and tags. Also add search for dataset description.
For field-level description and tags, we have two sources - one from ingest one from UI (editable). We simply add multiple search fields instead of merging them.
Checklist