Older newsreaders see new posts in GROUP, but can't actually list any

[NattyNarwhal]

This is neat; been meaning to write something like this for older systems. Curious how vintage NNTP clients handle it (since I'm curious about the retro applications), and so far, none of them like it:

• old Mozilla (but probably modern Thunderbird too) won't let you subscribe to groups that aren't listed already; if i use slrn to subscribe and add it to newsrc, it appears that it can see the metadata from GROUP, but can't request anything more; I don't see any further commands other than GROUP. No posts.
• Outlook Express suffers the same fate
• Xnews can see the group, but when it tries to get posts, it issues an XOVER command which nntpit rejects. It can at least add groups w/o them being listed.

[taviso]

Thanks for testing! I think adding XOVER support is no problem, I'll work on that.

Listing newsgroups is a problem because there's no API to do that. I could pre-seed it with the top 500 subreddits, so at least you would see something in those clients.

I can't imagine what the problem is with the GROUP command, maybe it's because the name doesn't match. The server is supposed to send:

211 count low high groupname

I'm sending back /r/groupname, because I didn't think it mattered.

[NattyNarwhal]

Removing the /r/ seems to make OE work. It issues an XOVER, then an XHDR (I assume the XOVER fails, and XHDR is the fallback?). OE seems to mention something about newsgroup descriptions as something you can enable, so perhaps the /r/ and subreddit metadata is better left there.

Mozilla still seems lost.

I also tried Opera M2 (going over what newsreaders I remember) and it fails on NEWGROUPS.

[NattyNarwhal]

FWIW a hokey NEWGROUPS could just ignore the args and return what LIST ACTIVE does:

diff --git a/nntpit.c b/nntpit.c
index 00e0cf2..3cc1798 100644
--- a/nntpit.c
+++ b/nntpit.c
@@ -727,6 +727,8 @@ void client_read(struct ev_loop *loop, ev_io *w, int revents)
 
             if (strcasecmp(cmd, "LIST") == 0) {
                 handle_list_cmd(cl, data);
+            } else if (strcasecmp(cmd, "NEWGROUPS") == 0) {
+                handle_list_cmd(cl, "ACTIVE");
             } else if (strcasecmp(cmd, "GROUP") == 0) {
                 handle_group_cmd(cl, data);
             } else if (strcasecmp(cmd, "HEAD") == 0) {

[taviso]

Okay, I added a quick newgroups stub. XOVER might take a little longer, but that should be fine too.

[taviso]

OH - I missed that you sent me a PR to do the same thing, sorry about that!

[NattyNarwhal]

FWIW, MicroPlanet Gravity has the same behaviour as Outlook Express (try XOVER, fall back to XHDR).

[maxcrees]

I looked into what seems to be preventing Thunderbird from working, other than listing groups (for that I just connected raw to nntpit and issued GROUP whatever, which then caused the group to show up in Thunderbird's dialog after a refresh).

It seems Thunderbird doesn't like if the low water mark is zero - it skips here from processing the articles. This results in a log like the following (MOZ_LOG=NNTP:5 thunderbird):

[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) opening connection to localhost on port 1119
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: NNTP_RESPONSE
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Receiving: 200 nntpit ready.
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: NNTP_LOGIN_RESPONSE
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: NNTP_SEND_MODE_READER
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Sending: MODE READER
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: NNTP_RESPONSE
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Receiving: 501 Unknown MODE.
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: NNTP_SEND_MODE_READER_RESPONSE
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: SEND_FIRST_NNTP_COMMAND
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Sending: GROUP adelielinux
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: NNTP_RESPONSE
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Receiving: 211 95 0 95 adelielinux
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: SEND_FIRST_NNTP_COMMAND_RESPONSE
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: SETUP_NEWS_STREAM
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: NNTP_XOVER_BEGIN
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) SetCurrentGroup to adelielinux
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: NNTP_FIGURE_NEXT_CHUNK
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Next state: NNTP_ERROR
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) ClosingConnection
[(null) 9154: Main Thread]: I/NNTP (0x55f50d6d0d80) Sending: QUIT

Note NNTP_FIGURE_NEXT_CHUNK followed by NNTP_ERROR. Correspondingly nntpit itself only shows:

[8] <- [MODE READER]                                                             
[8] <- [GROUP adelielinux]                                                       
[8] <- [QUIT]

The following very hacky patch allows Thunderbird to at least retrieve and display most articles, but there is some fencepost error where Thunderbird thinks that there is an article at <> (1) that doesn't exist.

diff --git a/nntpit.c b/nntpit.c
index 803c461..659a893 100644
--- a/nntpit.c
+++ b/nntpit.c
@@ -573,8 +573,8 @@ void handle_group_cmd(client_t *cl, const char *param)
 
     client_printf(cl, "211 %d %d %d %s\r\n",
         highwm - lowwm,
-        lowwm,
-        highwm,
+        lowwm + 1,
+        highwm + 1,
         param);
 
     client_flush(cl);
@@ -675,7 +675,7 @@ void handle_head_cmd(client_t *cl, const char *param, bool head, bool article)
         }
     } else {
         json_object_object_foreach(groupset, msg, artnum) {
-            if (json_object_get_int(artnum) == number) {
+            if (json_object_get_int(artnum) + 1 == number) {
 
                 // Now we lookup that id in the spool file.
                 if (!json_object_object_get_ex(spool, msg, &object)) {

Overall it needs you two who are much more knowledgeable about NNTP to take a look at, but that's at least the right direction I think.

This is with Thunderbird 68.9.0 for what it's worth.

[NattyNarwhal]

I think the 0 mark is reserved for if it doesn't know what an article is (like if you request headers with XOVER et al by message ID directly).

[lemtom]

The hacky patch allows me to retrieve posts in Thunderbird (after having used slrn to subscribe first). But every time I click on a message I get a prompt asking me "Would you like to subscribe to t3_pha2gy" (or some other id).

Is there any way to disable this in either Thunderbird of the code? I'm not sure what's going on with this (or if it is even related to the previous issue), but right now it seems to be the only remaining thing keeping nntpit from being comfortable to use for me.

[lemtom]

With the latest changes, the subscription prompt no longer seems to be an issue.

The threading hierarchy is quite messed up, but I assume that is caused by the hacky patch above (without which I am unfortunately still unable to retrieve messages).

[taviso]

I'll try out thunderbird today and see if I can figure out what's going on, it's possibly it's just being very strict about the syntax and we can make it work.

[taviso]

It seems to work okay for me in Thunderbird now. I setup a new news account, but there's no easy way to subscribe to unlisted group in Thunderbird.

You can do this instead:

$ echo retrotech: >> ~/.thunderbird/*/News/newsrc-reddit

[taviso]

(I think I was calculating the watermark incorrectly, and slrn is just very forgiving so I didn't notice!)

[lemtom]

Awesome! I indeed no longer require the hacky patch, and I seem to be getting the correct first messages of a hierarchy.

What I'm still getting now is that most threads are put under other other threads:

(screenshot of /r/belgium)

The threads in the screenshot:
https://old.reddit.com/r/belgium/comments/ttjjwd/belgium_on_the_1_of_april/
https://old.reddit.com/r/belgium/comments/ttlu91/rant_about_new_flight_tax/
https://old.reddit.com/r/belgium/comments/ttlu91/rant_about_new_flight_tax/

Gonna subscribe to a new subreddit, just in case this is just a relic from working with the hacky patch before.

[taviso]

Hmm - I see what you mean, that's confusing. The Xrefs all seem well-formed to me... I'll see if I can figure it out.