Missing GMail labels #48

ktf · 2017-11-15T07:14:06Z

Similarly to #46 I've:

pull: full synchronization (no previous synchronization state)                                                                                    [3/8741]
fetching messages: 123866it [04:10, 414.05it/s]
receiving content:   0%|                                                                             | 3/57916 [00:00<9:22:58,  1.71it/s]Traceback (most r
ecent call last):
  File "/nix/store/5zb6zlyxg2xjz7p3kqr9hahl3vdbzm63-python3.6-gmaileer-v0.3-20171115/bin/.gmi-wrapped", line 9, in <module>
    g.main ()
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 307, in pull
    self.full_pull ()
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 530, in full_pull
    updated = self.get_content (message_gids)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 595, in get_content
    self.remote.get_messages (need_content, _got_msg, 'raw')
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 282, in get_messages
    batch.execute (http = self.http)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/googleapiclient/http.py", line 1437, in execute
    self._callback(request_id, response, exception)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 262, in _cb
    cb (resp)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 593, in _got_msg
    self.local.store (m, db)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 346, in store
    self.update_tags (m, p, db)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 354, in update_tags
    labels = [self.gmailieer.remote.labels[l] for l in labels]
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 354, in <listcomp>
    labels = [self.gmailieer.remote.labels[l] for l in labels]
KeyError: 'Label_3'

I did not create a label while it was running and I do not have Label_3 as label.

The text was updated successfully, but these errors were encountered:

julian-klode · 2017-11-15T10:53:42Z

Just for the record: Labels have internal id like that, they are not necessarily the names you gave them. For example, I have a label ubuntu-devel, but it's internal id is Label_44. Here it's mapping the internal ids to the names AFAICT.

gauteh · 2017-11-15T11:07:34Z

Julian Andres Klode writes on november 15, 2017 11:53:

Just for the record: Labels have internal id like that, they are not necessarily the names you gave them. For example, I have a label ubuntu-devel, but it's internal id is Label_44. Here it's mapping the internal ids to the names AFAICT.

That's correct; we fetch the label list at start-up with all the mappings. You can print them with the '-t' option for `pull` I believe.. Does the relevant label show there? If you know what it is. Does re-starting the pull fix the issue? I don't know why not all the labels are pulled, could be an API issue.

ktf · 2017-11-15T15:08:41Z

gmi pull -t shows labels and the third one in the output order is FromMailingList, however I see no mapping output. Restarting the pull did not help. I do have somewhat strange labels though, like R&D or `Something/something - (else)" could that be a problem?

gauteh · 2017-11-16T10:10:28Z

Giulio Eulisse writes on november 15, 2017 16:18:

`gmi pull -t` shows labels and the third one in the output order is `FromMailingList`, however I see no mapping output. Restarting the pull did not help. I do have somewhat strange labels though, like `R&D` or `Something/something - (else)" could that be a problem?

I don't think so. I've added a change on master which also prints the gmail key. Can you see if you also get the problematic Label_3? - gaute

ktf · 2017-11-16T10:28:45Z

I don't get any Label_3:

+ gmi pull -t
+ grep Label_3
CWG4                           Label_3939416323915462435
OpenSource Mailing lists       Label_3194149110520196790
CMS                            Label_394156353969411168
Nix                            Label_3817195439730139360
Simulation & Generators        Label_3946793891174421901
Infrastructure                 Label_3990267443586131481
WP3 - Tools                    Label_3023913578609897416
Daily Build Reports            Label_330431641839543684
Banking                        Label_3461285147915119731

gauteh · 2017-11-16T10:31:33Z

Giulio Eulisse writes on november 16, 2017 11:28:

I don't get any `Label_3`: ``` + gmi pull -t + grep Label_3 CWG4 Label_3939416323915462435 OpenSource Mailing lists Label_3194149110520196790 CMS Label_394156353969411168 Nix Label_3817195439730139360 Simulation & Generators Label_3946793891174421901 Infrastructure Label_3990267443586131481 WP3 - Tools Label_3023913578609897416 Daily Build Reports Label_330431641839543684 Banking Label_3461285147915119731 ```

Do you have the chance to go here: https://developers.google.com/gmail/api/v1/reference/users/labels/list Enter your email in the example API on the right and post the result somewhere? I realize this is private, what I'm looking for is anything else than just labels.. According to the docs there shouldn't be any nextPage fields, but maybe you have enough labels..

ktf · 2017-11-16T10:46:11Z

Ciao,
I just did and AFAICT, there is no pagination entry in the output (nor `Label_3). I will try reducing the number of labels. How do you get Label_3 in the first place? Is it attached to the message?

gauteh · 2017-11-16T12:18:19Z

Giulio Eulisse writes on november 16, 2017 11:46:

Ciao, I just did and AFAICT, there is no pagination entry in the output (nor `Label_3). I will try reducing the number of labels. How do you get Label_3 in the first place? Is it attached to the message?

Yes, it comes from message.get().

gauteh · 2017-11-16T13:25:59Z

Did you identify which message has the Label_3? You could maybe add a print google-id so that you get some more output before it crashes (in Local.update_tags). Then you could try and fetch that message from the Gmail API explorer and see if the label is actually there as well. Can you test out labels.get() on Label_3 in the API explorer? Maybe it shows up if you try to fetch it directly..

ktf · 2017-11-16T13:46:46Z

For the moment I just patched it to add an unknown label in case the Remote label is not found. That should allow me to identify the broken messages as well. I want to get to the end and see how much I like notmuch before I spend time debugging this.

gauteh · 2017-11-16T13:49:05Z

Giulio Eulisse writes on november 16, 2017 14:46:

For the moment I just patched it to add an `unknown` label in case the Remote label is not found. That should allow me to identify the broken messages as well. I want to get to the end and see how much I like notmuch before I spend time debugging this.

Good plan. You might want to add unknown to ignored labels so that its not synced back up to your account if you do push or sync. Or just avoid pushing up before you've figured out the issue.

…

-g

ktf · 2017-11-16T13:55:22Z

Thanks for mentioning the ignored label stuff. I was actually wondering about that. I will add it. If I understand what's going on I'll let you know right away.

gauteh · 2017-11-16T13:56:10Z

Giulio Eulisse writes on november 16, 2017 14:55:

Thanks for mentioning the ignored label stuff. I was actually wondering about that. I will add it. If I understand what's going on I'll let you know right away.

Thanks, appreciate the effort.

ktf · 2017-11-16T14:11:32Z

Thank you. If this works as I expect it to work and scales to my mailbox size, you made an happy man.. ;-)

This adds a notmuch "unknown" tag whenever a label id which cannot be found in the list of available label ids is retrieved from a message. Notice the "unknown" tag is never synchronized.

ktf · 2017-11-16T21:43:14Z

With #49, I am able to go past my issue and I've a full sync done. Notice that some of the messages which are reported to have "Label_3" do not actually have any label associated. I wonder if this is simply a problem with some old labelling schema on google side..

gauteh · 2017-11-16T22:27:51Z

Weird; if you put in the messages in https://developers.google.com/gmail/api/v1/reference/users/messages/get do they show Label_3?

gauteh · 2017-11-19T11:26:49Z

Fixed in #51:

If you encounter this error run:

$ gmi set --drop-non-existing-labels

and re-try the pull.

This will drop any labels that cannot be found remotely. If the message tags are modified locally (through notmuch) the label will be removed on the remote side on the next push/sync.

ktf added a commit to ktf/gmailieer that referenced this issue Nov 16, 2017

Workaround gauteh#48

cd020bc

This adds a notmuch "unknown" tag whenever a label id which cannot be found in the list of available label ids is retrieved from a message. Notice the "unknown" tag is never synchronized.

gauteh changed the title ~~KeyError on first input~~ Missing GMail labels Nov 17, 2017

gauteh mentioned this issue Nov 17, 2017

Missing gmail label #51

Merged

gauteh added the gmail-api label Nov 17, 2017

gauteh closed this as completed Nov 19, 2017

gauteh added a commit that referenced this issue Dec 2, 2017

add note for workaround for missing labels (#48)

6d5ef69

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing GMail labels #48

Missing GMail labels #48

ktf commented Nov 15, 2017

julian-klode commented Nov 15, 2017

gauteh commented Nov 15, 2017 via email

ktf commented Nov 15, 2017

gauteh commented Nov 16, 2017 via email

ktf commented Nov 16, 2017

gauteh commented Nov 16, 2017 via email

ktf commented Nov 16, 2017

gauteh commented Nov 16, 2017 via email

gauteh commented Nov 16, 2017

ktf commented Nov 16, 2017

gauteh commented Nov 16, 2017 via email

ktf commented Nov 16, 2017

gauteh commented Nov 16, 2017 via email

ktf commented Nov 16, 2017

ktf commented Nov 16, 2017

gauteh commented Nov 16, 2017

gauteh commented Nov 19, 2017

Missing GMail labels #48

Missing GMail labels #48

Comments

ktf commented Nov 15, 2017

julian-klode commented Nov 15, 2017

gauteh commented Nov 15, 2017 via email

ktf commented Nov 15, 2017

gauteh commented Nov 16, 2017 via email

ktf commented Nov 16, 2017

gauteh commented Nov 16, 2017 via email

ktf commented Nov 16, 2017

gauteh commented Nov 16, 2017 via email

gauteh commented Nov 16, 2017

ktf commented Nov 16, 2017

gauteh commented Nov 16, 2017 via email

ktf commented Nov 16, 2017

gauteh commented Nov 16, 2017 via email

ktf commented Nov 16, 2017

ktf commented Nov 16, 2017

gauteh commented Nov 16, 2017

gauteh commented Nov 19, 2017