Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clarify what is a valid topic #137

Open
tomkralidis opened this issue May 6, 2024 · 16 comments · May be fixed by #146
Open

clarify what is a valid topic #137

tomkralidis opened this issue May 6, 2024 · 16 comments · May be fixed by #146
Labels
question Further information is requested

Comments

@tomkralidis
Copy link
Collaborator

As discussed with @golfvert, we need to clarify whether "partial" topics are deemed valid and can/should be used or not.

For example, origin/a/wis2/ca-eccc-msc/data/core/weather/surface-based-observations/synop is a valid topic.

Should origin/a/wis2/ca-eccc-msc/data/core/weather/surface-based-observations be considered a valid topic as well? This means a topic without a leaf?

We would need to update the specification to be clear in this regard (and whether Requirement 1B needs to be updated/augmented. In addition, we would need to update the artefacts made available on schemas.wmo.int for the WTH CV bundle/lookup.

@golfvert
Copy link
Contributor

golfvert commented May 6, 2024

For me, only "leaf" topics are usable. Otherwise origin/a/wis2/ca-eccc-msc/data/core/weather would be also usable...
As we have agreed sublevels in eg origin/a/wis2/ca-eccc-msc/data/core/weather/surface-based-observations then, one of them MUST be used.

@antje-s
Copy link
Contributor

antje-s commented May 6, 2024

I agree, if there are subtopics, one of the subtopics should be used, otherwise we will have many messages published to the top topics. If there are no suitable subtopic values (e.g. for "new" data), experimental should be used and the required new subtopic values should be added as soon as possible to main WTH. As soon as the new values are available, the publish must be switched back to the main branch of the WTH

@josusky
Copy link
Contributor

josusky commented May 6, 2024

Publishing only to "leafs" sounds good to me.

@amilan17
Copy link
Member

amilan17 commented May 6, 2024

I think there might be some use cases in prediction topic where they don't want to go down to the last leaf

@golfvert
Copy link
Contributor

golfvert commented May 6, 2024

Can you tell us which one ? Looking at what ECMWF is publishing I see only "leaf"

@amilan17
Copy link
Member

amilan17 commented May 6, 2024

Can you tell us which one ? Looking at what ECMWF is publishing I see only "leaf"

I don't see any examples on the explorer, but I think it would be good to double check with TT-NWPMD.

@tomkralidis
Copy link
Collaborator Author

Adding @sebvi for feedback / validation.

tomkralidis added a commit that referenced this issue Jun 3, 2024
@tomkralidis tomkralidis linked a pull request Jun 3, 2024 that will close this issue
@david-i-berry
Copy link
Member

What happens if there is a new CAP alert where none of the leaves are appropriate? If we do not allow publication on a non-leaf node we need to have 100 % coverage from day 1.

@amilan17
Copy link
Member

amilan17 commented Jun 3, 2024

If publication down to the last leaf is required, what should be done in the following scenario?

  1. data is already being published to topic .../discipline/topic-x/topic-a
  2. New sub-topics of topic-a are proposed .../discipline/topic-x/topic-a/1 and ...topic-x/topic-a/2

@amilan17
Copy link
Member

amilan17 commented Jun 3, 2024

Can you tell us which one ? Looking at what ECMWF is publishing I see only "leaf"

I don't see any examples on the explorer, but I think it would be good to double check with TT-NWPMD.

I spoke with Yuki and this will be on the agenda of the TT-NWPMD's next meeting.

@tomkralidis
Copy link
Collaborator Author

TT-WISMD 2024-06-03:

  • needs discussion with TT-NWPMD
  • need to verify for CAP workflow
  • how to we deal with change management of subtopics introduced against existing topics (communications)
  • FT implications? If new subtopics invalidate previous topics, this is not applicable for FT. Is this a breaking change then (i.e. INFCOM/EC approval)?
  • grey area for change management/migration
  • perhaps we can define rules for deprecation
  • a validation step against current traffic on GBs can help with gauging current activity against the impacted topic(s)
  • publishers can use experimental as an interim solution
  • subscribers can also be affected (wildcards can help mitigate)

@josusky
Copy link
Contributor

josusky commented Jun 3, 2024

I am still OK with publishing to leaf topics only.

Splitting of a leaf to multiple leaves is a change management issue that is largely unrelated.
We can propose some procedures that would make such change less of a "breaking change" than it may seem, such as:

  • recommend subscribers / data consumers to use wildcard (#) at the end of the subscribed topics
  • require the proposer of such a split to make a throw analysis to ensure that (1) there will be a matching sub-topic for all data currently published under the current leaf or there is no data at all, and (2) that the subscribers, if any, use the aforementioned wildcard - a GB could help with the latter, I guess

@sebvi
Copy link

sebvi commented Jun 14, 2024

Sorry for not responding back earlier. I don't see a use case where you would not publish up to the leaf.
We discussed this in our TT-NWPMD meetings and the reason we stopped at the hierarchy level 12 is because going further would become too specific and restrictive and could force institutions to publish at a branch level instead of the leaf.

I support publishing to leaf only

@tomkralidis
Copy link
Collaborator Author

TT-WISMD 2024-06-28:

  • how to manage the change
    • extending / adding leaves
    • adding a new subtopic invalidates the parent topic (breaking change)
    • for ESDs with no subtopics yet, can they use experimental?
      • may be risky (could turn experimental into a free-for-all)
    • provide a transition period between two FTs
  • perhaps too early to address in the lifespan of WIS2

ACTION: @tomkralidis to raise at WIS2 Architecture.

@amilan17 amilan17 modified the milestones: FT2024-2, Future release Jul 19, 2024
@golfvert
Copy link
Contributor

Did we achieve a conclusion on this ?

@tomkralidis
Copy link
Collaborator Author

We will revisit / discuss at the next TT-WISMD meeting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment