Hangups fails to find all contact names

[xmikos]

I get KeyError in on_connect() method in __main__.py when hangups tries to get user names for user_ids in conversations:

first_names = [contacts[user_ids]['first_name']
                        for user_ids in
                        conversations[conv_id]['participants']]

It works if I change it to:

first_names = [contacts[user_ids]['first_name'] if user_ids in contacts else "UNKNOWN"
                        for user_ids in
                        conversations[conv_id]['participants']]

But then I have a lot of UNKNOWN user names in list of conversations.

[tdryer]

Have a look at the code that parses the contacts:
https://github.com/tdryer/hangups/blob/master/hangups/client.py#L257

The format is pretty strange, so it's likely missing some contacts.

[eternaleye]

That commit actually causes a similar failure mode for me - one of my conversations seems to have neither a first_name nor a full_name associated with it, so both https://github.com/tdryer/hangups/blob/master/hangups/client.py#L142 and the line after it fail with KeyErrors on HEAD = ce88f81

That said, this is awesome work!

EDIT: Okay, so looking at the mobile Hangouts app, the failing conversation is a contact who I never talked to via the official Hangouts app - only via bitlbee/XMPP. Could that be it?

[tdryer]

Thanks for trying it out!

Can you post an excerpt from hangups.log? There should be a line like this, followed by the response hangups is trying to parse:
DEBUG:hangups.client:Response to request for contacts/getentitybyid was 200:

[eternaleye]

Well, this is that message (my sister, amusingly enough):

(snipped, see EDIT2)

However, that does have the display_name and first_name fields.
I suspect it's misattributing the error, since if I do the
if 'first_name' in entity['properties'] else 'UNKNOWN' (and ditto display_name)
trick, then the UNKNOWN is not with her (and there's a group chat with her
and my dad which loads fine)

EDIT: Oh, I see. There's multiple entities in the response, and the one that's actually at fault is not my sister.
EDIT2: Since I put the json of the entry that broke things below, I'm going to remove the full dump, since it has my sister's email and such in it.

[eternaleye]

If I json.dumps the failed entity, it's this:
{"blocked": false, "id": {"chat_id": "108087445024636237743", "gaia_id": "108087445024636237743"}, "properties": {"type": "NONE"}, "entity_type": "GAIA"}

[tdryer]

Right, it appears to be looking up two users at once, and one of them is "type": "NONE" instead of "type": "ES_USER".

[eternaleye]

Ooooh, I think I know what happened. I think that's someone who deleted their G+ account

[tdryer]

Good catch. If you can guess who that is, the question is do they appear in the Hangouts app and does it know their name somehow?

[eternaleye]

They do and it does - the hangouts app on Android orders conversations the same way as your TUI client, which makes it easy. Just find the two conversations that bracket it.

[tdryer]

Can you add a logger.debug(user_ids_list) to the beginning the of the get_users function? This is a list of tuples with two IDs for each user. Usually the IDs are identical, but in this case I'm wondering if they could be different.

[eternaleye]

No, looks like both IDs are the same:
{('100614481081814380012', '100614481081814380012'), ('108087445024636237743', '108087445024636237743')}

(the second tuple being the broken one, and the first being my sister)

[tdryer]

If you're interested, the next step is to watch the traffic when the Hangouts Chrome extension loads to find out how it gets the name. You can press F5 while in the Chrome developer tools network tab to force the extension to reload and capture the requests.

[eternaleye]

Ah, I don't even have Chrome installed. Firefox user here :P

[tdryer]

It should be possible to accomplish the same thing with Firefox's developer tools and the Hangouts widget in Gmail.

[tdryer]

ce88f81 should handle this now so users with no name are called UNKNOWN.

[eternaleye]

It looks like it comes as part of the big Javascript-in-HTML dump from https://talkgadget.google.com/u/0/talkgadget/_/chat ; working on exactly what the format is.

[eternaleye]

(or rather, where in data_dict it winds up)

[eternaleye]

Aha! Currently, on client.py:429 you use p[0] as the tuple of user_ids - p[1] is the name of the user!

[eternaleye]

If I move

self._initial_contacts = {}

up, and add

full_name = p[1]
self._initial_contacts[user_ids] = {
    'first_name': full_name.split(r'\s+')[0],
    'full_name': full_name,
}

into the participants loop, it works perfectly, with the contact names overwriting the 'temporary' participant ones if found.

[tdryer]

Interesting idea, pulling the names from the conversation rather than the contact. I could almost do without all the complicated contact parsing code. Too bad it doesn't provide the first_name separately.

[eternaleye]

Eh, the contact parsing code is valuable for when you add the ability to invite additional participants/start new chats

[tdryer]

I committed that workaround. Thanks again!

[eternaleye]

No problem! Glad I was able to track this down.

[Equidamoid]

I still have crashes due to this issue, maybe it should put some stub name ("Unknown"?) and save the iser when p[1] is None in the workaround code?

[tdryer]

Can you paste the stacktrace?

[Equidamoid]

  File "hangups/hangups/client.py", line 479, in _init_talkgadget_1
    ) for conv_id, conv_info,in initial_conversations.items()}
  File "hangups/hangups/client.py", line 479, in <dictcomp>
    ) for conv_id, conv_info,in initial_conversations.items()}
  File "hangups/hangups/client.py", line 475, in <listcomp>
    for user_id
KeyError: UserID(chat_id='...', gaia_id='...')

this code solves the issue:

                if len(p) > 1 and p[1]:
                    display_name = p[1]
                else:
                    display_name = "Unknown"
                    self.initial_users[user_id] = User(
                        id_=user_id, first_name=display_name.split()[0],
                        full_name=display_name,
                        is_self=(user_id == self.self_user_id)
                    )

[tdryer]

I pushed a commit to implement that workaround. Let me know if it works.

[Equidamoid]

It works!

[tdryer]

Great! Thanks for trying it out.

[endofline]

we've implemented a workaround for the "unknown user" issue by populating a global _user_list within our bot, then using getentitybyid to refresh users that show up as unknown. the list update occurs when the bot initialises and on group chat membership change. of course it would be better that the hangups library can refresh its own list, but my lack of experience in asyncio makes it a bit hard to modify the hangups client itself to actually do this.

[endofline]

i've "merged" the workaround previously posted into a hangups fork for testing purposes, the latest commit being this: endofline@3726795 (https://github.com/endofline/hangups/tree/workaround/fallback-user)

[tdryer]

Thanks @endofline!

I've added a new function build_user_list that constructs a UserList and takes care of calling getentitybyid for the missing users. With this change there should be no more unknowns (at least at startup).

[endofline]

@tdryer i just pulled and built the latest master, the user list still seems to contain "Unknown" , but no longer has "unknown". do you require additional logs for debugging?

[endofline]

my mistake again @tdryer - didn't examine the change at hangups/ui/__main__.py, which for the record is:

        self._user_list = hangups.UserList(self._client,
                                           initial_data.self_entity,
                                           initial_data.entities,
                                           initial_data.conversation_participants)

to

        self._user_list = yield from hangups.user.build_user_list(
            self._client, initial_data
        )

And the _on_connect() is now a coroutine as well.

Since the bot is based on the reference client, we will also have to follow suit.

After applying the changes, it appears that only 1 Unknown shows up now - a person with a deleted G+ profile

[tdryer]

Do you get any "Adding fallback User" warnings any more? I'm curious whether that can be removed now.

Unfortunately I don't have any contacts with a deleted G+ profile, so I can't debug that.

[endofline]

No more "adding fallback user"s - the userlist appears clean now (except for that missing g+ profile), haven't quite tested membership changes yet

[wakest]

is there a way to force an already installed version of hangups to re "build_user_list"? about 90% of my contacts show up as just the phone numbers. running the latest version of hangups.

[tdryer]

The user list is rebuilt every time hangups is started. If those are Google Voice contacts, you're probably affected by #52.

[avaidyam]

@wakest @tdryer Any GVoice contacts (that is, phone numbers) won't show up because Hangouts secretly queries Google's People API to populate their data. The People API is what is supposed to replace the old Contacts API and is currently get-only and returns a subset of your contacts. I would suggest looking into that API to get information about a phone number, or, if developing a local client and there's an API for local contacts, query that.

[EionRobb]

@avaidyam I do this in the purple-hangouts plugin, but then there's a weird cascading priority of display names - do you take the ones in the People/Contacts API or the "get entity by id" list or the Hangouts conversation object fallback name as authoritative? Gets messy.

[avaidyam]

@EionRobb You'd prioritize the People API, only for the GVoice, and let the Hangouts API handle GAIA entities (basically, Google+ profiles).

I've noticed other chat clients having this issue as well, but the GAIA stuff really comes down to preferring Google+ profile settings or a user's personal contacts information. A user may or may not correctly format contact data and it may or may not be synchronized with Google+ or Facebook. It's a policy you'd need to think about.