category slots belief state error

[tsinghuarj]

When reviewing bad cases(model prediction VS ground truth)，i found a number of dialogues exists belief state error: 1_00105、4_00020、4_00000、3_00112、1_00029、6_00062、3_00115,etc. mainly category slots error.
Take 3_00115 for example (a fragment as below), number_of_baths slot should be "1", but the belief state missed.
What I am concerned about is that if a certain percentage of errors are made, it may affect the effectiveness of the model.

Can you check this?

{
        "frames": [
          {
            "slots": [
              {
                "exclusive_end": 51,
                "slot": "property_name",
                "start": 39
              },
              {
                "exclusive_end": 93,
                "slot": "address",
                "start": 70
              },
              {
                "exclusive_end": 154,
                "slot": "rent",
                "start": 148
              }
            ],
            "actions": [
              {
                "slot": "property_name",
                "act": "OFFER",
                "values": [
                  "Fremont Arms"
                ]
              },
              {
                "slot": "address",
                "act": "OFFER",
                "values": [
                  "37811 Fremont Boulevard"
                ]
              },
              {
                "slot": "number_of_beds",
                "act": "OFFER",
                "values": [
                  "1"
                ]
              },
              {
                "slot": "number_of_baths",
                "act": "OFFER",
                "values": [
                  "1"
                ]
              },
              {
                "slot": "rent",
                "act": "OFFER",
                "values": [
                  "$2,000"
                ]
              },
              {
                "slot": "count",
                "act": "INFORM_COUNT",
                "values": [
                  "7"
                ]
              }
            ],
            "service": "Homes_1"
          }
        ],
        "utterance": "I've found 7 apartments. One is called Fremont Arms and is located at 37811 Fremont Boulevard. The apartment has 1 bed room, 1 bath room, and costs $2,000 per month.",
        "speaker": "SYSTEM"
      },
      {
        "frames": [
          {
            "state": {
              "active_intent": "FindApartment",
              "slot_values": {
                "pets_allowed": [
                  "True"
                ],
                "number_of_beds": [
                  "1"
                ],
                "area": [
                  "Fremont"
                ],
                "property_name": [
                  "Fremont Arms"
                ]
              },
              "requested_slots": []
            },
            "slots": [],
            "service": "Homes_1"
          }
        ],
        "utterance": "That one sounds good.",
        "speaker": "USER"
      }

[XiaoxueZang]

Thanks for your question. We checked the examples you listed here and they are labeled as intended. In the two turns from dialogue 3_00115 in the dev set pasted above, the user’s requirement is to find an apartment with 1 bedroom in Fremont. When the user agrees with a property proposed by the system, we interpret that it agrees to the specific entity but not necessarily all the attributes of that entity such as number_of_bathroom, furnished, pets_allowed etc, so we only include property_name in the state. If the user explicitly says that "I want the house to only have one bathroom" instead of something like "sounds good", we’ll include number_of_bathroom = 1 in the dialogue state. Hope this helps.

[tsinghuarj]

Thanks for your reply. I still have question on this issue.

1. in dialogue 3_00112 i mentioned above , the user express '3 attached baths',but the state missed

{
        "frames": [
          {
            "service": "Homes_1",
            "slots": [
              {
                "exclusive_end": 47,
                "slot": "area",
                "start": 37
              }
            ],
            "state": {
              "active_intent": "FindApartment",
              "requested_slots": [],
              "slot_values": {
                "area": [
                  "Sebastopol"
                ],
                "number_of_beds": [
                  "3"
                ]
              }
            }
          }
        ],
        "speaker": "USER",
        "utterance": "Please can you find some good one in Sebastopol and I would prefer to have three bed rooms with 3 attached baths."
      }

2. As to dialogue 3_00115, i think it is a little confused. When system offer a slot-value property_name(which in required_slots ), and user confirmed, then the state update; but when system offer a slot-value number_of_baths(which in optional_slots), and user confirmed, the state not update? I think it's difficult for decide which slot should update unless we have a priori that 'we interpret that it agrees to the specific entity but not necessarily all the attributes of that entity ', but on semantic level dialogue flow(Since the conversation have been generated by 'Building a Conversational Agent Overnight with Dialogue Self-Play', i think semantic level is enough for system turn), i can not see this priori. So my opinion is that when system offer a required/optional slot-value , and user confirmed, the state should update. Unless there is a priori in system semantic representation, say, offer property_name, inform number_of_baths. In that case, it's reasonable not update number_of_baths state.

[XiaoxueZang]

1. Thanks for pointing it out. We just corrected it.
2. This dataset is generated using a new approach which will be released after the competition. The user may negate the entity or change its preferences in the following turns and any extra information in the dialogue states will constrain the entities that the system could possibly offer. How dialogue state is updated when the user accepts the system's offer is designed to be consistent across the whole dataset. Hope it helps answer your question.

[tsinghuarj]

Thank you very much for your patience.
I think i got it.
And a suggestion as below:
You want the model to learn which slot offered by system is confirmed by user using context , this problem is raised by system but not user. In fact the model do not need to learn this if we have a better system semantic level design.

[XiaoxueZang]

Thanks for your suggestion.