Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing GMail labels #48

Closed
ktf opened this issue Nov 15, 2017 · 17 comments
Labels

Comments

@ktf
Copy link
Contributor

@ktf ktf commented Nov 15, 2017

Similarly to #46 I've:

pull: full synchronization (no previous synchronization state)                                                                                    [3/8741]
fetching messages: 123866it [04:10, 414.05it/s]
receiving content:   0%|                                                                             | 3/57916 [00:00<9:22:58,  1.71it/s]Traceback (most r
ecent call last):
  File "/nix/store/5zb6zlyxg2xjz7p3kqr9hahl3vdbzm63-python3.6-gmaileer-v0.3-20171115/bin/.gmi-wrapped", line 9, in <module>
    g.main ()
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 307, in pull
    self.full_pull ()
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 530, in full_pull
    updated = self.get_content (message_gids)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 595, in get_content
    self.remote.get_messages (need_content, _got_msg, 'raw')
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 282, in get_messages
    batch.execute (http = self.http)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/googleapiclient/http.py", line 1437, in execute
    self._callback(request_id, response, exception)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 262, in _cb
    cb (resp)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 593, in _got_msg
    self.local.store (m, db)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 346, in store
    self.update_tags (m, p, db)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 354, in update_tags
    labels = [self.gmailieer.remote.labels[l] for l in labels]
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 354, in <listcomp>
    labels = [self.gmailieer.remote.labels[l] for l in labels]
KeyError: 'Label_3'

I did not create a label while it was running and I do not have Label_3 as label.

@julian-klode

This comment has been minimized.

Copy link
Contributor

@julian-klode julian-klode commented Nov 15, 2017

Just for the record: Labels have internal id like that, they are not necessarily the names you gave them. For example, I have a label ubuntu-devel, but it's internal id is Label_44. Here it's mapping the internal ids to the names AFAICT.

@gauteh

This comment has been minimized.

Copy link
Owner

@gauteh gauteh commented Nov 15, 2017

@ktf

This comment has been minimized.

Copy link
Contributor Author

@ktf ktf commented Nov 15, 2017

gmi pull -t shows labels and the third one in the output order is FromMailingList, however I see no mapping output. Restarting the pull did not help. I do have somewhat strange labels though, like R&D or `Something/something - (else)" could that be a problem?

@gauteh

This comment has been minimized.

Copy link
Owner

@gauteh gauteh commented Nov 16, 2017

@ktf

This comment has been minimized.

Copy link
Contributor Author

@ktf ktf commented Nov 16, 2017

I don't get any Label_3:

+ gmi pull -t
+ grep Label_3
CWG4                           Label_3939416323915462435
OpenSource Mailing lists       Label_3194149110520196790
CMS                            Label_394156353969411168
Nix                            Label_3817195439730139360
Simulation & Generators        Label_3946793891174421901
Infrastructure                 Label_3990267443586131481
WP3 - Tools                    Label_3023913578609897416
Daily Build Reports            Label_330431641839543684
Banking                        Label_3461285147915119731
@gauteh

This comment has been minimized.

Copy link
Owner

@gauteh gauteh commented Nov 16, 2017

@ktf

This comment has been minimized.

Copy link
Contributor Author

@ktf ktf commented Nov 16, 2017

Ciao,
I just did and AFAICT, there is no pagination entry in the output (nor `Label_3). I will try reducing the number of labels. How do you get Label_3 in the first place? Is it attached to the message?

@gauteh

This comment has been minimized.

Copy link
Owner

@gauteh gauteh commented Nov 16, 2017

@gauteh

This comment has been minimized.

Copy link
Owner

@gauteh gauteh commented Nov 16, 2017

Did you identify which message has the Label_3? You could maybe add a print google-id so that you get some more output before it crashes (in Local.update_tags). Then you could try and fetch that message from the Gmail API explorer and see if the label is actually there as well. Can you test out labels.get() on Label_3 in the API explorer? Maybe it shows up if you try to fetch it directly..

@ktf

This comment has been minimized.

Copy link
Contributor Author

@ktf ktf commented Nov 16, 2017

For the moment I just patched it to add an unknown label in case the Remote label is not found. That should allow me to identify the broken messages as well. I want to get to the end and see how much I like notmuch before I spend time debugging this.

@gauteh

This comment has been minimized.

Copy link
Owner

@gauteh gauteh commented Nov 16, 2017

@ktf

This comment has been minimized.

Copy link
Contributor Author

@ktf ktf commented Nov 16, 2017

Thanks for mentioning the ignored label stuff. I was actually wondering about that. I will add it. If I understand what's going on I'll let you know right away.

@gauteh

This comment has been minimized.

Copy link
Owner

@gauteh gauteh commented Nov 16, 2017

@ktf

This comment has been minimized.

Copy link
Contributor Author

@ktf ktf commented Nov 16, 2017

Thank you. If this works as I expect it to work and scales to my mailbox size, you made an happy man.. ;-)

ktf added a commit to ktf/gmailieer that referenced this issue Nov 16, 2017
This adds a notmuch "unknown" tag whenever a label id which cannot be found
in the list of available label ids is retrieved from a message. Notice the
"unknown" tag is never synchronized.
@ktf

This comment has been minimized.

Copy link
Contributor Author

@ktf ktf commented Nov 16, 2017

With #49, I am able to go past my issue and I've a full sync done. Notice that some of the messages which are reported to have "Label_3" do not actually have any label associated. I wonder if this is simply a problem with some old labelling schema on google side..

@gauteh

This comment has been minimized.

Copy link
Owner

@gauteh gauteh commented Nov 16, 2017

Weird; if you put in the messages in https://developers.google.com/gmail/api/v1/reference/users/messages/get do they show Label_3?

@gauteh gauteh changed the title KeyError on first input Missing GMail labels Nov 17, 2017
@gauteh gauteh added the gmail-api label Nov 17, 2017
@gauteh

This comment has been minimized.

Copy link
Owner

@gauteh gauteh commented Nov 19, 2017

Fixed in #51:

If you encounter this error run:

$ gmi set --drop-non-existing-labels

and re-try the pull.

This will drop any labels that cannot be found remotely. If the message tags are modified locally (through notmuch) the label will be removed on the remote side on the next push/sync.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.