Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Statistics fail to update #10508

Closed
2 tasks done
joubu opened this issue Nov 29, 2023 · 19 comments · Fixed by #10528
Closed
2 tasks done

Statistics fail to update #10508

joubu opened this issue Nov 29, 2023 · 19 comments · Fixed by #10528
Assignees
Labels
bug Something is broken.
Milestone

Comments

@joubu
Copy link
Contributor

joubu commented Nov 29, 2023

Describe the issue

The background job updating the stats is failing with the following error:

Failure while executing task: KeyError: 'stats_timestamp'

I already tried

  • I've read and searched the documentation.
  • I've searched for similar issues in this repository.

Steps to reproduce the behavior

Update a translation.

Expected behavior

No response

Screenshots

No response

Exception traceback

weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,727: ERROR/1511] Failure while executing task: KeyError: 'stats_timestamp'                                                              weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,727: ERROR/ForkPoolWorker-8] Failure while executing task: KeyError: 'stats_timestamp'                                                  
weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,728: ERROR/1511] Failure while executing task: KeyError: {'signal': <Signal: task_failure providing_args={'exception', 'task_id', 'traceback', 'args', 'kwargs', 'einfo'}>, 'sender': <@task: weblate.utils.tasks.update_translation_stats_parents of weblate at 0x7fc9e3c49210>, 'task_id': '41c6e00c-36ec-4f25-99c4-35b7e471fa56', 'args': [7621], 'kwargs'
: {}, 'traceback': <traceback object at 0x7fc9cd31e240>, 'einfo': <ExceptionInfo: ExceptionWithTraceback()>}                                                                                                         weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,728: ERROR/ForkPoolWorker-8] Failure while executing task: KeyError: {'signal': <Signal: task_failure providing_args={'exception', 'task
_id', 'traceback', 'args', 'kwargs', 'einfo'}>, 'sender': <@task: weblate.utils.tasks.update_translation_stats_parents of weblate at 0x7fc9e3c49210>, 'task_id': '41c6e00c-36ec-4f25-99c4-35b7e471fa56', 'args': [7621], 'kwargs': {}, 'traceback': <traceback object at 0x7fc9cd31e240>, 'einfo': <ExceptionInfo: ExceptionWithTraceback()>}                                                                                             weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,728: ERROR/1511] Failure while executing task                                                                                           weblate-docker-weblate-1       | celery-celery stderr | Traceback (most recent call last):                                                                                                                           weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task                                                                weblate-docker-weblate-1       | celery-celery stderr |     R = retval = fun(*args, **kwargs)                                                                                                                        weblate-docker-weblate-1       | celery-celery stderr |                  ^^^^^^^^^^^^^^^^^^^^                                                                                                                        weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__                                                        weblate-docker-weblate-1       | celery-celery stderr |     return self.run(*args, **kwargs)                                                                                                                         
weblate-docker-weblate-1       | celery-celery stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                         weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/tasks.py", line 79, in update_translation_stats_parents                                        
weblate-docker-weblate-1       | celery-celery stderr |     translation.stats.update_parents()                                                                                                                       weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 312, in update_parents                                                         
weblate-docker-weblate-1       | celery-celery stderr |     stat.update_stats()                                                                                                                                      weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 329, in update_stats                                                           
weblate-docker-weblate-1       | celery-celery stderr |     self.calculate_basic()                                                                                                                                   weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 337, in calculate_basic                                                        
weblate-docker-weblate-1       | celery-celery stderr |     self._calculate_basic()                                                                                                                                  weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 827, in _calculate_basic                                                       
weblate-docker-weblate-1       | celery-celery stderr |     stats[item] = max(values, default=time.time())                                                                                                           weblate-docker-weblate-1       | celery-celery stderr |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                           
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 824, in <genexpr>                                                              weblate-docker-weblate-1       | celery-celery stderr |     values = (stats_obj.aggregate_get(item) for stats_obj in all_stats)                                                                                      
weblate-docker-weblate-1       | celery-celery stderr |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                  weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 225, in aggregate_get                                                          weblate-docker-weblate-1       | celery-celery stderr |     return self._data[name]                                                                                                                                  weblate-docker-weblate-1       | celery-celery stderr |            ~~~~~~~~~~^^^^^^                                                                                                                                  
weblate-docker-weblate-1       | celery-celery stderr | KeyError: 'stats_timestamp'                                                                                                                                  weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,728: ERROR/ForkPoolWorker-8] Failure while executing task                                                                               
weblate-docker-weblate-1       | celery-celery stderr | Traceback (most recent call last):                                                                                                                           weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task                                                                
weblate-docker-weblate-1       | celery-celery stderr |     R = retval = fun(*args, **kwargs)                                                                                                                        weblate-docker-weblate-1       | celery-celery stderr |                  ^^^^^^^^^^^^^^^^^^^^                                                                                                                        
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__                                                        weblate-docker-weblate-1       | celery-celery stderr |     return self.run(*args, **kwargs)                                                                                                                         
weblate-docker-weblate-1       | celery-celery stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                         weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/tasks.py", line 79, in update_translation_stats_parents                                        
weblate-docker-weblate-1       | celery-celery stderr |     translation.stats.update_parents()                                                                                                                       weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 312, in update_parents                                                         
weblate-docker-weblate-1       | celery-celery stderr |     stat.update_stats()
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 329, in update_stats
weblate-docker-weblate-1       | celery-celery stderr |     self.calculate_basic()
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 337, in calculate_basic
weblate-docker-weblate-1       | celery-celery stderr |     self._calculate_basic()
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 827, in _calculate_basic
weblate-docker-weblate-1       | celery-celery stderr |     stats[item] = max(values, default=time.time()) 
weblate-docker-weblate-1       | celery-celery stderr |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 824, in <genexpr>
weblate-docker-weblate-1       | celery-celery stderr |     values = (stats_obj.aggregate_get(item) for stats_obj in all_stats)
weblate-docker-weblate-1       | celery-celery stderr |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 225, in aggregate_get
weblate-docker-weblate-1       | celery-celery stderr |     return self._data[name]
weblate-docker-weblate-1       | celery-celery stderr |            ~~~~~~~~~~^^^^^^
weblate-docker-weblate-1       | celery-celery stderr | KeyError: 'stats_timestamp'

How do you run Weblate?

Docker container

Weblate versions

weblate:bleeding (48863d1)

Weblate deploy checks

No response

Additional context

Related to #10473

@nijel nijel self-assigned this Nov 29, 2023
@nijel nijel added this to the 5.3 milestone Nov 29, 2023
@nijel nijel added the bug Something is broken. label Nov 29, 2023
@nijel nijel closed this as completed in bee1808 Nov 29, 2023
@joubu
Copy link
Contributor Author

joubu commented Nov 29, 2023

Thanks a lot!

Copy link

Thank you for your report; the issue you have reported has just been fixed.

  • In case you see a problem with the fix, please comment on this issue.
  • In case you see a similar problem, please open a separate issue.
  • If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

@joubu
Copy link
Contributor Author

joubu commented Nov 30, 2023

Even with the fix from #10508 (using bleeding docker image with HEAD 2f6cb95) we are still seeing:
image
image

And clicking the "1707 unfinished" link leads us to "No strings found!"

@nijel
Copy link
Member

nijel commented Nov 30, 2023

The stats will update only upon translation change, not automatically.

@joubu
Copy link
Contributor Author

joubu commented Nov 30, 2023

Yes, I have updated a translation, also flag one as "need editing". But still marked as 90% translated.

@joubu
Copy link
Contributor Author

joubu commented Nov 30, 2023

None of the "Unfinished" number is correct.
https://translate.koha-community.org/projects/koha/master/#languages

French: 1731, but if you click on this number you get 2090
Spanish: 1707, 0
German: 1709, 217
Greek: 9673, 14752
and so on...

@nijel
Copy link
Member

nijel commented Nov 30, 2023

Do you still see some errors in the log? I've just pushed 3537b3d so that stats age is visible on the page, that would give us a clue whether it is wrongly calculated or update was skipped.

@joubu
Copy link
Contributor Author

joubu commented Nov 30, 2023

No error found in the log.

I will wait for the next docker image to be build and I will update!

@joubu
Copy link
Contributor Author

joubu commented Nov 30, 2023

Another weirdness:
https://translate.koha-community.org/projects/koha/23-05/#languages (another category: "23-05" instead of "master").

German is 100% translated, it has the green tick. On hover text is: "1859 strings".
On the component list view: https://translate.koha-community.org/projects/koha/23-05/-/de/
1859 is the number for the first component "installer".

This "installer" component has the "source code repo" linked to the remote git repo, whereas other components within this 23-05 category point to it using weblate://koha/23-05/installer

@joubu
Copy link
Contributor Author

joubu commented Dec 1, 2023

Debug commit applied (docker bleeding 423bbe5)

Numbers are still wrong.

Master/Spanish components have been updated "6 hours ago"
https://translate.koha-community.org/projects/koha/master/installer/es/#information
Same for 23-05/German
https://translate.koha-community.org/projects/koha/23-05/installer/de/#information

Note that "0" is displayed when at the category level (but I guess it's expected):
https://translate.koha-community.org/projects/koha/23-05/-/de/#information

@nijel
Copy link
Member

nijel commented Dec 1, 2023

0 is not expected, that means that the stats were not updated with the new code yet. So there is clearly an issue with updating category/language stats.

nijel added a commit to nijel/weblate that referenced this issue Dec 1, 2023
These were missing for some time so these were never updated.

Fixes WeblateOrg#10508
@nijel
Copy link
Member

nijel commented Dec 1, 2023

#10528 should address this.

nijel added a commit to nijel/weblate that referenced this issue Dec 1, 2023
These were missing for some time so these were never updated.

Fixes WeblateOrg#10508
nijel added a commit that referenced this issue Dec 1, 2023
These were missing for some time so these were never updated.

Fixes #10508
@laoneo
Copy link

laoneo commented Dec 4, 2023

Mounted the patch from #10528 into my docker instance and it is now correctly updating.

@joubu
Copy link
Contributor Author

joubu commented Dec 5, 2023

Everything is fixed and working perfectly! Thanks again!

@laoneo
Copy link

laoneo commented Dec 19, 2023

Should this patch be in 5.3 (docker)? I'v updated and still have the issue, it only disappears when I mount the patched file with the fix from here.

@nijel
Copy link
Member

nijel commented Dec 19, 2023

The patch is there, but there is another issue which prevents component wide stat updates from being propagated. That one is fixed in c77ca8b.

@laoneo
Copy link

laoneo commented Dec 19, 2023

Ok, thanks

@nijel nijel modified the milestones: 5.3, 5.3.1 Dec 19, 2023
nijel added a commit that referenced this issue Dec 19, 2023
@nijel
Copy link
Member

nijel commented Dec 19, 2023

I've just released 5.3.1 with the fix. Docker images will follow shortly.

@laoneo
Copy link

laoneo commented Dec 20, 2023

Thanks, can confirm that the fix is working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is broken.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants