New emails are not marked as seen when fetched with curl

[orycho]

I did this

I am unsing libcurl to fetch unseen emails from an account. Until curl version 7.55 (as far as I know) emails where then marked as read and where not foud again. This has changed by version 7.66 and probably still persists.

I expected the following

Emails should be marked as read once fetched

curl/libcurl version

7.66.0
[curl -V output]
Not provided. Instead, you can reproduce by using this program repeatedly on an email account with an unread email:
imap-fetch-new.c.txt

operating system

Linux kinshasa 5.9.8 #1 SMP Mon Nov 16 21:02:53 CET 2020 x86_64 x86_64 x86_64 GNU/Linux

Thanks,
Ory

[bagder]

I don't know what the problem is so I can't fix this. Isn't this app making curl issue a UID FETCH command? What else would curl do?

[orycho]

Hi, curl also used to have the emails marked as read after the fetch. This seems to no longer ge the case. That is why the same uid is found by "search unseen" after being fetched. Thanks, Ory--Diese Nachricht wurde von meinem Android Mobiltelefon mit GMX Mail gesendet.Am 27.09.21, 10:03 schrieb Daniel Stenberg ***@***.***>: I don't know what the problem is so I can't fix this. Isn't this app making curl issue a UID FETCH command? What else would curl do? —You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

[bagder]

You already said that in the original description. I don't know what that means. curl once used FETCH directly without UID, but it never explicitly marked anything as "read" as far as I can remember.

[orycho]

Well, I tried to make it reproducible with the program in the attachment. I have the hunch that the fetch request is not performed until the very end somehow? But with an older curl version (7.59 iirc) the emails get marked as seen on the mail server... --Diese Nachricht wurde von meinem Android Mobiltelefon mit GMX Mail gesendet.Am 27.09.21, 10:09 schrieb Daniel Stenberg ***@***.***>: You already said that in the original description. I don't know what that means. curl once used FETCH directly without UID, but it never explicitly marked anything as "read" as far as I can remember. —You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

[bagder]

Well, I tried to make it reproducible with the program in the attachment.

Sure, but I'm saying I don't know what the right behavior would be, IMAP wise, so I cannot fix this.

I have the hunch that the fetch request is not performed until the very end somehow?

You're getting the email, right? Then the fetch command must've been performed. Or rather the UID FETCH command to be specific.

[orycho]

Okay, maybe "completed" would have been the better word. Still, I would expect the server to mark the email as seen after the fetch, and with an older curl version that does happen... --Diese Nachricht wurde von meinem Android Mobiltelefon mit GMX Mail gesendet.Am 27.09.21, 10:16 schrieb Daniel Stenberg ***@***.***>: Well, I tried to make it reproducible with the program in the attachment. Sure, but I'm saying I don't know what the right behavior would be, IMAP wise, so I cannot fix this. I have the hunch that the fetch request is not performed until the very end somehow? You're getting the email, right? Then the fetch command must've been performed. Or rather the UID FETCH command to be specific. —You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

[orycho]

I'm not at work this week, so I cannot check. but with the older curl version I believe there comes a last message from the mail server that says the seen flag was applied and that is no longer the case.--Diese Nachricht wurde von meinem Android Mobiltelefon mit GMX Mail gesendet.Am 27.09.21, 10:16 schrieb Daniel Stenberg ***@***.***>: Well, I tried to make it reproducible with the program in the attachment. Sure, but I'm saying I don't know what the right behavior would be, IMAP wise, so I cannot fix this. I have the hunch that the fetch request is not performed until the very end somehow? You're getting the email, right? Then the fetch command must've been performed. Or rather the UID FETCH command to be specific. —You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

[bagder]

In this case, it doesn't help to know what an older curl did.

[orycho]

Well, do you have a suggestion how I can get the seen email to be marked as seen? With a quotes command perhaps? --Diese Nachricht wurde von meinem Android Mobiltelefon mit GMX Mail gesendet.Am 27.09.21, 10:32 schrieb Daniel Stenberg ***@***.***>: In this case, it doesn't help to know what an older curl did. —You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

[bagder]

If I knew what the correct way would be I would not say that I don't know...

[orycho]

Okay, guess I will have to experiment once I'm back in the office --Diese Nachricht wurde von meinem Android Mobiltelefon mit GMX Mail gesendet.Am 27.09.21, 10:36 schrieb Daniel Stenberg ***@***.***>: If I knew what the correct way would be I would not say that I don't know... —You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

[mogery]

UID FETCH should automatically set the \Seen flag server-side. If you're getting any e-mails back through it, they should be marked as seen.

[mogery]

7.66.0 [curl -V output] Not provided. Instead, you can reproduce by using this program repeatedly on an email account with an unread email: imap-fetch-new.c.txt

This code only issues SEARCH commands, which are not supposed to set the \Seen flag. This is correct behaviour.
If you want the \Seen flag to be set, you will need to issue a UID FETCH command.

[bagder]

thanks a lot for this feedback @mogery

It seems to suggest that it works as intended.

[jay]

So it would seem this is a server issue, not a curl issue? Can we close this then?

[jay]

Can we close this then?

Oops... meant to actually ask the question, not be snarky about it! Reopened.

[mogery]

So it would seem this is a server issue, not a curl issue? Can we close this then?

It's user error. Seems closeable to me.

[jay]

It's user error. Seems closeable to me.

Ok thanks. Closing.

[orycho]

I'm actually not happy about this as the same software I have in produciton behaves correctly with an old libcurl and incorrectly with a later one. So I would rule out a server issue.
My feeling is that there is a last bit of communication missing: the older version reported a last "flag /seen" before it was finished with the email fetch... but if nobody knows what causes that then we're stuck anyways.

[mogery]

I'm actually not happy about this as the same software I have in produciton behaves correctly with an old libcurl and incorrectly with a later one. So I would rule out a server issue. My feeling is that there is a last bit of communication missing: the older version reported a last "flag /seen" before it was finished with the email fetch... but if nobody knows what causes that then we're stuck anyways.

That would be odd, because SEARCH commands do not implicitly set IMAP \Seen flags. That means, that with the code you linked, the current behaviour is the correct behaviour.

[orycho]

That example was specifically constructed as a minimal reproduciton example.
It fetches the email on lines 122 to 130. Fetch command is printed out on line 123 an performed on 130.
If you execute the code multiple times on a mail account with an unread mail, it should fetch and print the same email each time.
That was consistent with the versions I list in the original description.

[mogery]

That example was specifically constructed as a minimal reproduciton example. It fetches the email on lines 122 to 130. Fetch command is printed out on line 123 an performed on 130. If you execute the code multiple times on a mail account with an unread mail, it should fetch and print the same email each time. That was consistent with the versions I list in the original description.

Oh, my apologies. I've misread the code. This is still pretty odd though.

What is the version of the old libcurl that the code was working on?

[orycho]

"I did this

I am using libcurl to fetch unseen emails from an account. Until curl version 7.55 (as far as I know) emails where then marked as read and where not foud again. This has changed by version 7.66 and probably still persists."

So, libcurl 7.55 is the version where it works.

[jay]

I am using libcurl to fetch unseen emails from an account. Until curl version 7.55 (as far as I know) emails where then marked as read and where not foud again. This has changed by version 7.66 and probably still persists.

The confusion here might be that curl has switched its message retrieval mode from message sequence numbers to UID numbers.

For example

if you SEARCH UNSEEN and it returns 11970 that's a message sequence number.

but then you query via url ;UID=11970 but that's a UID number. if you turn on verbose mode in both you'll see any recent curl send UID FETCH 11970 BODY[], whereas older versions would send FETCH 11970 BODY[].

Try UID SEARCH UNSEEN to get the UID numbers, then pass that.

edit: Alternately since 7.62.0 you can use ;MAILINDEX=11970 to retrieve body via message sequence number however that is more error prone because message sequence numbers are relative and may be reassigned during a session.

[mogery]

Try UID SEARCH UNSEEN to get the UID numbers, then pass that

+1

If you take a look at the output of the repro code, the mails that the code is printing out are ones that are actually already read.

[orycho]

Okay, that is something I an try. Thanks, I'll let you know tomorrow if that helped!

Cheers!

[orycho]

Hi, the UID SEARCH UNSEEN seems to do the trick. I thought I was doing UID only code, maybe the documentation could be reviewed? But that aside it's all good now.

Thanks for the help!