recent releases are missing on the site #18

oshanz · 2021-07-20T02:32:48Z

as you can see in here, according to the site latest snapshot is 20210712.
however tumbleweed cli list more recent releases.

boombatower · 2021-07-21T01:11:53Z

Yeah, I reported the issue in #opensuse-admin IRC channel, but never received a response. Not sure if it's me or openSUSE.

$ ./src/main.py mail -s 2017-12
git-sync: Preparing. Repo in .git /root/.cache/tumbleweed-review/tumbleweed-review-site
git-sync: Mode sync
git-sync: Using origin/master
git-sync: Fetching from origin/master
Warning: Permanently added the RSA host key for IP address '140.82.114.4' to the list of known hosts.
From github.com:boombatower/tumbleweed-review-site
 * branch              master     -> FETCH_HEAD
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
git-sync: In sync, all fine.
[I] ingest 2021-07
Traceback (most recent call last):
  File "./src/main.py", line 74, in <module>
    ret = args.func(args)
  File "/srv/src/mail.py", line 317, in argparse_main
    main(args.logger, cache_dir, args.start_month, output_dir, not args.no_refresh)
  File "/srv/src/mail.py", line 298, in main
    mbox_paths = mboxes_download(cache_dir, start_month, refresh)
  File "/srv/src/mail.py", line 91, in mboxes_download
    shutil.copyfileobj(mbox_gzip, mbox_file)
  File "/usr/lib64/python3.6/shutil.py", line 79, in copyfileobj
    buf = fsrc.read(length)
  File "/usr/lib64/python3.6/gzip.py", line 276, in read
    return self._buffer.read(size)
  File "/usr/lib64/python3.6/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/usr/lib64/python3.6/gzip.py", line 482, in read
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached

The URL being downloaded works in a browser for me and I have not explored any further

https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/export/factory@lists.opensuse.org-2021-07.mbox.gz?start=2021-07-01&end=2021-08-01

Started occurring Jul 14, 2021 11:05am CDT. Any thoughts would be helpful.

boombatower · 2021-07-21T01:13:05Z

Perhaps it would be useful for me to expose my CI setup so anyone could see when things are not working and why.

boombatower · 2021-07-21T02:37:51Z

Appears to be a real bug on mailing list site.

curl 'https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/export/factory@lists.opensuse.org-2021-07.mbox.gz?start=2021-07-01&end=2021-08-01' --output 2021-07.gz

Results in a mailbox file ending with:

Any feedback (code re

From an email with the subject New package: python-keyring-keyutils

My email client shows that email last line as the following.

Any feedback (code reviews, bug reports etc.) is always welcome!:)

With a signature and referenced URLs afterwards.

For whatever reason curl and browser do not complain about the file ending early, but the problem appears to be genuine. This should be fixed upstream by mailing list host/software, but I could also add error checking to process the partial file.

boombatower · 2021-07-21T03:00:47Z

Created https://progress.opensuse.org/issues/95756

alexandru-m-g · 2021-07-25T19:10:31Z

I don't know if this is related or relevant but the graphs from here https://metrics.opensuse.org/d/osrt_release/osrt-release?orgId=1 also end on 2021-07-12

boombatower · 2021-07-25T20:45:46Z

Those graphs ingest data from the review site, so yes related.

temporarium · 2021-07-26T19:27:21Z

Waiting patiently for your excellent service to be up and running again...

Anything others can do to help get it moving?

boombatower · 2021-07-26T19:29:29Z

Resolve the progress.opensuse.org issue covering the data corruption from the mailing list server. I could add an exception handler to keep processing, but then it would be presenting partial information which I think is worse. As soon as the p.o.o issue is resolve the CI run should update the site.

boombatower · 2021-07-27T22:30:31Z

Added a link to this bug on the review site.

temporarium · 2021-07-28T12:06:36Z

Added a link to this bug on the review site.

Saw the link yesterday. Don't see it anymore. Does that mean it shall soon be back in operation?

alexandru-m-g · 2021-08-05T14:25:22Z

Any timeframe from opensuse about fixing the mailing list ? I have my work computer on Tumbleweed (I know, not advisable but was working great in connection with the https://review.tumbleweed.boombatower.com/ service ).

boombatower · 2021-08-12T03:21:48Z

No update on progress.opensuse.org issue in regard to mailing list fix.

cgoetzke · 2021-08-17T02:07:53Z

Didn't you have a problem with the mailing list archives omitting information a while back? And didn't it take the openSUSE folks a good while to get that fixed?

Given this issue has been sitting untouched by openSUSE for 26 days now, I'm not holding my breath for a fix.

Honestly, it almost feels like someone at openSUSE is hostile to this project, and I can't for the life of me figure out why.

temporarium · 2021-08-17T11:42:01Z

https://openqa.opensuse.org/ appears stuck as well... Maybe they're all on vacation.

boombatower · 2021-08-20T22:34:22Z

Yes, and bugzilla before that. Apparently openSUSE infra team could use help.

thorgrin · 2021-08-24T15:12:13Z

This might be related to https://gitlab.com/mailman/hyperkitty/-/issues/383 fixed by https://gitlab.com/mailman/hyperkitty/-/merge_requests/347
However, if this is the case, the already imported messages would need to be reimported to enable the download again.

The problem probably happened at 14.7.2021 since https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/export/factory@lists.opensuse.org-2021-07.mbox.gz?start=2021-07-01&end=2021-07-13 works as expected, but https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/export/factory@lists.opensuse.org-2021-07.mbox.gz?start=2021-07-14&end=2021-08-01 returns only a 10 byte long file. Continuing from 15.7.2021, everything is fine again.

Therefore, a temporary workaround can probably be implemented to download on a daily basis and just skip (and report) the broken days.

JoeSalmeri · 2021-09-02T15:45:38Z

Therefore, a temporary workaround can probably be implemented to download on a daily basis and just skip (and
report) the broken days

That seems like a great idea...flag the days where a problem exists with "bad data" or something like that and then continue to calculate for days where the problem does not exist.

I am surprised that this issue has been broken now for 6 or 7 weeks.

physxP · 2021-09-09T12:54:35Z

Is there any other way of getting the stability info? I need to change snapshot on my work PC to more stable version.

ericjs · 2021-09-09T17:37:08Z

FWIW in experimenting with these urls the 2021-07-1 to 2021-08-01 one that Throgrin posted reliably downloads an empty mbox file (in a 10 byte gzip), the 2021-07-01 to 2021-07-13 one he posted seems to reliably download a complete mbox file. The original 2021-07-14 to 2021-08-01 one from boombatower behaves non-deterministically. Each time I try it it truncates the mbox file in a slightly different place, and at least once I managed to get one that didn't seem to be truncated (or perhaps it was just not noticeably so).

ericjs · 2021-09-09T17:39:02Z

Is there any other way of getting the stability info? I need to change snapshot on my work PC to more stable version.

How about requesting archives via the mail interface itself?

ericjs · 2021-09-09T19:03:05Z

Is the syntax of that export URL documented anywhere?

boombatower · 2021-09-17T04:29:57Z

Is the syntax of that export URL documented anywhere?

I believe someone told me the params in discord so not sure.

Feel free to submit a PR, but rather busy right now.

My main concern with skipping day data is we have no way in the display to indicate days are being skipped which means we could skip right over problem threads. If someone wants to implement the day ingest (for the month of issue and beyond) I'll try and review.

thorgrin · 2021-09-17T07:13:54Z

My main concern with skipping day data is we have no way in the display to indicate days are being skipped which means we could skip right over problem threads. If someone wants to implement the day ingest (for the month of issue and beyond) I'll try and review.

I believe that the problem was fixed in the hyperkitty and should be available in the next release, according to their changelog. Therefore, it should solve itself in the future.

I guess that It's better to have incomplete information than none - right now I'm updating my tumbleweed distros to the latest release and just hoping for the best.

Maybe, we could just ignore that and continue with August, as that month seems to download fine again?

boombatower · 2021-10-03T19:10:28Z

[I] ingest 2021-10
Traceback (most recent call last):
  File "./src/main.py", line 74, in <module>
    ret = args.func(args)
  File "/srv/src/mail.py", line 322, in argparse_main
    main(args.logger, cache_dir, args.start_month, output_dir, not args.no_refresh)
  File "/srv/src/mail.py", line 303, in main
    mbox_paths = mboxes_download(cache_dir, start_month, refresh)
  File "/srv/src/mail.py", line 96, in mboxes_download
    shutil.copyfileobj(mbox_gzip, mbox_file)
  File "/usr/lib64/python3.6/shutil.py", line 79, in copyfileobj
    buf = fsrc.read(length)
  File "/usr/lib64/python3.6/gzip.py", line 276, in read
    return self._buffer.read(size)
  File "/usr/lib64/python3.6/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/usr/lib64/python3.6/gzip.py", line 482, in read
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached

Apparently, very much not fixed on openSUSE list.

ericjs · 2021-11-03T15:54:56Z

Definitely still a problem. FWIW, the gzip is not getting truncated but rather the .mbox file within, which is why curl and the browser don't complain. Even date ranges that appear to work seem to suffer the same problem but (I would guess) get lucky with only harmless whitespace being truncated. If I do multiple downloads (with curl or browser) with the same date range and look at the size in bytes of the resultant file, that size is different every time, even with ranges that appear to produce a complete file.

ericjs · 2021-11-03T16:00:17Z

BTW is there any way to draw attention to the bug? I've posted my observations there, and while I know nothing about the code involved, the sort of behavior I described should surely make it obvious where to look in the code for the problem.

tneo · 2022-01-21T20:02:15Z

@boombatower the ticket contains a potential solution. Have you tried that? Is it possible to use https://openqa.opensuse.org/ for the quality rating, based upon the amount of tests passed or not

JoeSalmeri · 2022-01-23T15:46:23Z

@tneo

That's the method I have been using to determine which build to update to since this problem appears to be stuck in limbo.

It has worked out well for me.

temporarium · 2022-01-27T23:38:36Z

@tneo

That's the method I have been using to determine which build to update to since this problem appears to be stuck in limbo.

It has worked out well for me.

Same here. I only update to builds with low failure rates. So far, so good.

boombatower · 2023-07-28T04:11:16Z

Since this was mentioned recently I decided to revive things and see if the mailing list is fixed...

[I] ingest 2023-07
...
EOFError: Compressed file ended before the end-of-stream marker was reached

Huge progress, much bug.

digitaltrails · 2023-10-16T06:38:31Z

Would it be feasible/workable to just the mailing list and procmail (or similar) arriving mail it to a local archive?

Recently Tumbleweed has sent more papercuts my way than usual. Some kind of quality metric is sorely missed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

recent releases are missing on the site #18

recent releases are missing on the site #18

oshanz commented Jul 20, 2021

boombatower commented Jul 21, 2021

boombatower commented Jul 21, 2021

boombatower commented Jul 21, 2021

boombatower commented Jul 21, 2021

alexandru-m-g commented Jul 25, 2021

boombatower commented Jul 25, 2021

temporarium commented Jul 26, 2021

boombatower commented Jul 26, 2021

boombatower commented Jul 27, 2021

temporarium commented Jul 28, 2021

alexandru-m-g commented Aug 5, 2021

boombatower commented Aug 12, 2021

cgoetzke commented Aug 17, 2021

temporarium commented Aug 17, 2021

boombatower commented Aug 20, 2021

thorgrin commented Aug 24, 2021 •

edited

JoeSalmeri commented Sep 2, 2021

physxP commented Sep 9, 2021

ericjs commented Sep 9, 2021

ericjs commented Sep 9, 2021

ericjs commented Sep 9, 2021

boombatower commented Sep 17, 2021

thorgrin commented Sep 17, 2021

boombatower commented Oct 3, 2021

ericjs commented Nov 3, 2021

ericjs commented Nov 3, 2021 •

edited

tneo commented Jan 21, 2022

JoeSalmeri commented Jan 23, 2022

temporarium commented Jan 27, 2022

boombatower commented Jul 28, 2023

digitaltrails commented Oct 16, 2023

recent releases are missing on the site #18

recent releases are missing on the site #18

Comments

oshanz commented Jul 20, 2021

boombatower commented Jul 21, 2021

boombatower commented Jul 21, 2021

boombatower commented Jul 21, 2021

boombatower commented Jul 21, 2021

alexandru-m-g commented Jul 25, 2021

boombatower commented Jul 25, 2021

temporarium commented Jul 26, 2021

boombatower commented Jul 26, 2021

boombatower commented Jul 27, 2021

temporarium commented Jul 28, 2021

alexandru-m-g commented Aug 5, 2021

boombatower commented Aug 12, 2021

cgoetzke commented Aug 17, 2021

temporarium commented Aug 17, 2021

boombatower commented Aug 20, 2021

thorgrin commented Aug 24, 2021 • edited

JoeSalmeri commented Sep 2, 2021

physxP commented Sep 9, 2021

ericjs commented Sep 9, 2021

ericjs commented Sep 9, 2021

ericjs commented Sep 9, 2021

boombatower commented Sep 17, 2021

thorgrin commented Sep 17, 2021

boombatower commented Oct 3, 2021

ericjs commented Nov 3, 2021

ericjs commented Nov 3, 2021 • edited

tneo commented Jan 21, 2022

JoeSalmeri commented Jan 23, 2022

temporarium commented Jan 27, 2022

boombatower commented Jul 28, 2023

digitaltrails commented Oct 16, 2023

thorgrin commented Aug 24, 2021 •

edited

ericjs commented Nov 3, 2021 •

edited