requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: #68

vserbu · 2022-02-22T18:30:18Z

For some reason, some of the twitter account result in error, and previously those accounts worked. Last one is @Ultrekillblast and before that it was official @startrek account. I got this:

...
ℹ 2022-02-22 18:23:31,681 - pleroma_bot - INFO - tweets gathered: 27
ℹ 2022-02-22 18:23:34,148 - pleroma_bot - INFO - tweets to post: 27
ℹ 2022-02-22 18:23:34,150 - pleroma_bot - INFO - (1/27)
✖ 2022-02-22 18:23:34,402 - pleroma_bot - ERROR - Exception occurred (cli.py:502)
Traceback (most recent call last):
File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 103, in post_pleroma
response.raise_for_status()
File "/home/mastodon/.local/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://salocha.online/api/v1/media

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 482, in main
post_id = user.post_pleroma(
File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 117, in post_pleroma
response.raise_for_status()
File "/home/mastodon/.local/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://salocha.online/api/v1/media
...

robertoszek · 2022-02-23T06:44:57Z

Hi @vserbu !
What version of pleroma-bot are you using?

vserbu · 2022-02-23T07:54:42Z

I'm still on the 1.0.1 version. How do I update on newer version?

robertoszek · 2022-02-23T07:56:26Z

If you installed it using pip, you can upgrade to the latest version using this command:

pip install pleroma-bot --upgrade

vserbu · 2022-02-23T08:39:28Z

I did it, but the error is the same. It starts to post tweets and then blocks on one (in that example it was on second one) and that is it. Is there any cache to clear or something? I tried to limit the size of the tweet because I taught maybe it could be some big video or image, but it is not.

robertoszek · 2022-02-23T08:44:23Z

Would you mind sharing the output of your second try?

And maybe also your current config file (with the tokens or any sensitive info removed).

vserbu · 2022-02-23T08:54:29Z

Yes, I'm seeing it is somewhat different:

...
ℹ 2022-02-23 08:53:03,014 - pleroma_bot - INFO - Processing user: 2139
ℹ 2022-02-23 08:53:04,529 - pleroma_bot - INFO - tweets gathered: 4
ℹ 2022-02-23 08:53:05,787 - pleroma_bot - INFO - tweets to post: 4
ℹ 2022-02-23 08:53:05,787 - pleroma_bot - INFO - (1/4)
✖ 2022-02-23 08:53:05,930 - pleroma_bot - ERROR - Exception occurred
Unprocessable Entity
Validation failed: File content type is invalid, File is invalid
File: /home/mastodon/tweets/1496229640171630596/0.mp4 (_pleroma.py:127)
⚠ 2022-02-23 08:53:05,931 - pleroma_bot - WARNING - Error uploading media: 422 (_pleroma.py:134)
✖ 2022-02-23 08:53:05,984 - pleroma_bot - ERROR - Exception occurred (cli.py:536)
Traceback (most recent call last):
File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 513, in main
post_id = user.post_pleroma(
File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 167, in post_pleroma
response.raise_for_status()
File "/home/mastodon/.local/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://salocha.online/api/v1/statuses
...

robertoszek · 2022-02-23T09:29:09Z

It looks like the Mastodon instance is probably reporting the wrong type for this attachment:
https://video.twimg.com/tweet_video/FMOs2sNXMAEWLxe.mp4

$ file 0.mp4 
0.mp4: ISO Media

Hence, failing the validation. The instance wrongly expects an ISO (.iso) file (or something else than an .mp4) to be uploaded based on running the file utility.

pleroma-bot can't do much here, the issue lies on the instance's file command reporting the wrong type. It could be that it's an old version and updating to a more recent version would fix it.

This shouldn't block you from using the bot normally, for these cases we skip the upload of the problematic media and try to publish the tweet's text anyway.
But in this case, the tweet's text is empty and it only has the attachment, so it also fails to post an empty status.

robertoszek · 2022-02-23T10:08:45Z

I've published a release candidate version which won't fail due to the empty status (it will skip posting it), so at least it's not a fatal error and you could continue using the bot for publishing the rest of the tweets if you so choose.
If you want to try it you can install it like so:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc2

However, the file issue will still occur, the problematic attachment won't be uploaded because the Mastodon instance won't recognize it as an mp4 media file, there's not much the bot can do about that.

vserbu · 2022-02-23T10:45:35Z

I'll give it a try, tnx :)

vserbu · 2022-02-23T10:51:59Z

It's working, but I have a new problem. Visibility is suddenly not public even dough I stated visibility: "public" already. Now the new tweets are not in LOCAL timeline, only in my HOME timeline.

robertoszek · 2022-02-23T11:02:41Z

Nice catch!
I somehow missed an edge-case where the visibility is defined as a global value, sorry about that!
Feel free to give it a try with 1.0.3rc3:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc3

vserbu · 2022-02-23T11:44:43Z

Still the same problem :(

robertoszek · 2022-02-23T11:49:05Z

Really? That's odd, I thought I had found the issue.
Could you share your current config? (without the tokens or sensitive info)

vserbu · 2022-02-23T14:54:00Z

pleroma_base_url: https://salocha.online
max_tweets: 40
twitter_token: xxxxx
users:
- twitter_username: TheSpaceshipper
  pleroma_username: 1771
  pleroma_token: xxx
- twitter_username: TrekNewsNet
  pleroma_username: 1792
  pleroma_token: xxx
- twitter_username: retroscifiart
  pleroma_username: 1808
  pleroma_token: xxx
- twitter_username: 70sscifi
  pleroma_username: 1812
  pleroma_token: xxx
- twitter_username: tosgifs
  pleroma_username: 1833
  pleroma_token: xxx
- twitter_username: ultrakillblast
  pleroma_username: 2139
  pleroma_token: xxx
visibility: "public"
include_replies: false
file_max_size: 5MB
include_rts: false

vserbu · 2022-02-23T15:30:45Z

It looks like it's working now!

robertoszek · 2022-02-23T16:27:57Z

Great!
I was about to say that I could not seem to be able to reproduce the visibility issue on my side with your config on 1.0.3rc3 haha

Maybe I should've clarified that it only would apply to new posts published after upgrading to 1.0.3rc3 (the ones published already would've kept the wrong visibility).

I'll include these fixes on the next stable release as well.
Have a nice day!

robertoszek · 2022-02-23T16:32:13Z

And thank you for finding these issues and reporting them as well!

vserbu · 2022-02-23T17:08:04Z

Thank you for quick response :)

vserbu · 2022-02-23T20:06:35Z

I think we have another Issue regarding this.
The script stuck at:

ℹ 2022-02-23 20:03:35,112 - pleroma_bot - INFO - config path: /home/mastodon/config.yml 
ℹ 2022-02-23 20:03:35,113 - pleroma_bot - INFO - tweets temp folder: /home/mastodon/tweets 
ℹ 2022-02-23 20:03:35,119 - pleroma_bot - INFO - ====================================== 
ℹ 2022-02-23 20:03:35,120 - pleroma_bot - INFO - Processing user:       1771 
ℹ 2022-02-23 20:03:38,831 - pleroma_bot - INFO - tweets gathered:        18 
Processing tweets...  |

And that is it....stuck.

When I removed that account (#) everything else went ok.

robertoszek · 2022-02-24T05:50:29Z

I may have an idea of what's happening there.
Does running it with this version make any difference?

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc5

vserbu · 2022-02-24T10:09:37Z

It is not stucked any more but we are back with the similar error as on the beggining.


ℹ 2022-02-24 10:01:19,351 - pleroma_bot - INFO - config path: /home/mastodon/config.yml 
ℹ 2022-02-24 10:01:19,351 - pleroma_bot - INFO - tweets temp folder: /home/mastodon/tweets 
ℹ 2022-02-24 10:01:19,364 - pleroma_bot - INFO - ====================================== 
ℹ 2022-02-24 10:01:19,364 - pleroma_bot - INFO - Processing user:       1771 
ℹ 2022-02-24 10:01:21,926 - pleroma_bot - INFO - tweets gathered:        34 
ℹ 2022-02-24 10:01:24,416 - pleroma_bot - INFO - tweets to post:         7 
ℹ 2022-02-24 10:01:24,417 - pleroma_bot - INFO - (1/7) 
✖ 2022-02-24 10:01:24,871 - pleroma_bot - ERROR - Exception occurred (cli.py:537) 
Traceback (most recent call last):
  File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 514, in main
    post_id = user.post_pleroma(
  File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 170, in post_pleroma
    response.raise_for_status()
  File "/home/mastodon/.local/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://salocha.online/api/v1/statuses

robertoszek · 2022-02-24T13:11:15Z

Hmm... I'm wondering if you're hitting Mastodon's character limit (500 per toot) with a quoted tweet.

Would you mind testing out my theory by running this version?:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc6

vserbu · 2022-02-24T14:45:00Z

It's working, you were absolutely - right! tnx!

robertoszek · 2022-02-24T14:47:22Z

Nice!
I should be the one thanking you :)
Thanks to your reporting we found these issues and the fixes will be included on the next stable release.
Cheers!

nemobis · 2022-04-03T15:04:06Z

Hmm... I'm wondering if you're hitting Mastodon's character limit (500 per toot) with a quoted tweet.

When truncating a toot, it might be worth truncating before the signature, to avoid breaking links: see https://respublicae.eu/@Europarl_DE/108068732034451913 for an example.

I see that's actually what you were trying to do, not sure what went wrong:
8f44c53#diff-46774f4895c169807f7a3d85cc4ebc34a18ac37e7f4b2ed7306d5ef6977e59adR156

robertoszek · 2022-04-03T17:22:59Z

When truncating a toot, it might be worth truncating before the signature, to avoid breaking links: see https://respublicae.eu/@Europarl_DE/108068732034451913 for an example.

@nemobis
Oh, right! I missed how this would happen when a signature and original date are both enabled.
it should be fixed on 1.0.3rc8, feel free to give it a spin and see if the changes solved it:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc8

nemobis · 2022-04-03T18:17:09Z

Il 03/04/22 20:23, robertoszek ha scritto:

it should be fixed on 1.0.3rc8, feel free to give it a spin and see if the changes solved it:

Yes, thank you! All good on 1.0.3rc9.

nemobis · 2022-04-10T06:42:07Z

I'm still seeing some HTTP 422. Consistently for this one account:

ℹ 2022-04-10 09:29:33,942 - pleroma_bot - INFO - Processing user:       108070048904140658
ℹ 2022-04-10 09:29:36,909 - pleroma_bot - INFO - tweets gathered:        49
ℹ 2022-04-10 09:29:44,172 - pleroma_bot - INFO - tweets to post:         49
ℹ 2022-04-10 09:29:44,173 - pleroma_bot - INFO - (1/49)
✖ 2022-04-10 09:29:56,582 - pleroma_bot - ERROR - Exception occurred for user, skipping... (cli.py:556) 
Traceback (most recent call last):
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/cli.py", line 536, in main
    tweet["possibly_sensitive"],
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_utils.py", line 605, in post
    post_id = self.post_pleroma(tweet, poll, sensitive)
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_pleroma.py", line 170, in post_pleroma
    response.raise_for_status()
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://respublicae.eu/api/v1/statuses

I think it might be one of these tweets (cw: controversial political content):

robertoszek · 2022-04-10T20:39:54Z

@nemobis
For those specific 2 tweets you provided, I've not been able to reproduce the issue, they posted fine on a Mastodon instance on my testing (with or without signature/original date too).

Maybe the culprit is some other tweet.
You can try running the bot in verbose mode and see if it helps to narrow it down.

nemobis · 2022-04-10T21:14:31Z

Il 10/04/22 23:40, robertoszek ha scritto:

Maybe the culprit is some other tweet.

Yeah I don't know, I'll add some logging locally to find out.

nemobis · 2022-04-22T16:53:28Z

I keep getting HTTP 422 for https://respublicae.eu/@EP_Defence possibly due to https://nitter.eu/EP_Defence/status/1513526853532897283#m . Could it be because there are 4+2 images there considering the quoted tweet?

robertoszek · 2022-04-26T18:38:29Z

I keep getting HTTP 422 for https://respublicae.eu/@EP_Defence possibly due to https://nitter.eu/EP_Defence/status/1513526853532897283#m . Could it be because there are 4+2 images there considering the quoted tweet?

@nemobis Right! Nice catch, Mastodon only allows 4 images per post, possibly that's the cause. (Misskey and Pleroma allow 16).

I've added the upper limit for Mastodon to 1.0.3rc13, feel free to try it out and see if you still get an HTTP 422 for the same tweet:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc13

nemobis · 2022-07-29T22:05:29Z

Some quote-tweets with 5+ photos are still getting an HTTP 422 on the most recent release, though I'm not quite sure why (I even hacked the Mastodon code to raise the limit to 8).

Culprit seems to be https://nitter.eu/andreykovatchev/status/1534110522722996224#m

ℹ 2022-07-30 01:02:44,385 - pleroma_bot - INFO - Processing user:       108069143892397143
ℹ 2022-07-30 01:02:46,464 - pleroma_bot - INFO - tweets gathered:        7
ℹ 2022-07-30 01:02:50,080 - pleroma_bot - INFO - tweets to post:         7
ℹ 2022-07-30 01:02:50,081 - pleroma_bot - INFO - (1/7)
✖ 2022-07-30 01:02:57,280 - pleroma_bot - ERROR - Exception occurred for user, skipping... (cli.py:574) 
Traceback (most recent call last):
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/cli.py", line 554, in main
    tweets_to_post["media_processed"]
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_utils.py", line 620, in post
    post_id = self.post_pleroma(tweet, poll, sensitive, media)
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_pleroma.py", line 192, in post_pleroma
    response.raise_for_status()
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://respublicae.eu/api/v1/statuses

nemobis · 2022-07-29T22:28:39Z

On the other hand a bunch of posts which previously failed are now succeeding, like https://respublicae.eu/@Cedefop/108732982556276877 from https://nitter.eu/Cedefop/status/1542082206641840129 . 🤷

nemobis · 2022-11-18T08:25:10Z

I'm getting an HTTP 422 while trying to post (I think) this https://nitter.it/NatalieAmiri/status/1586320608538902528#m , a tweet with a video quoting another tweet with a video.

It would be easier to debug if the exception printed the original tweet ID too.

nemobis · 2022-11-18T11:06:00Z

A RT https://nitter.it/EmmanuelMacron/status/1591869166046380035#m failed too, maybe any quote tweet with video fails?

robertoszek · 2022-11-18T16:06:13Z

Ah, for the first tweet you mentioned it's probably because I don't think Mastodon supports multiple videos on the same post:
https://user-images.githubusercontent.com/1080963/202745994-93c67a8f-417e-465c-9f0a-0f69c61e7085.png

Hmm I need to think what's the best way to handle that. I assume it's better to drop the video on the quoted/retweeted tweet instead of the one attached to the user you're mirroring. You'll lose some context either way, which is a shame.

Ah, I see it's the same problem for the other tweet too (multiple videos):
https://user-images.githubusercontent.com/1080963/202748467-169040d2-e560-4b2d-9a31-ff607e7035a7.png

I'll open a new issue to keep track of it, as the "422 Unprocessable Entity" error is too generic and could be multiple things causing it.

nemobis · 2022-11-18T16:27:13Z

Il 18/11/22 18:06, robertoszek ha scritto:

I'll open a new issue to keep track of it, as the "422 Unprocessable Entity" error is too generic and could be multiple things causing it.

Thanks!

robertoszek self-assigned this Feb 23, 2022

robertoszek added the bug Something isn't working label Feb 23, 2022

robertoszek mentioned this issue Jun 6, 2022

Bump to v1.1.0 #87

Merged

robertoszek closed this as completed in #87 Jun 6, 2022

requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: #68

requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: #68

Comments

vserbu commented Feb 22, 2022

robertoszek commented Feb 23, 2022

vserbu commented Feb 23, 2022

robertoszek commented Feb 23, 2022

vserbu commented Feb 23, 2022

robertoszek commented Feb 23, 2022

vserbu commented Feb 23, 2022

robertoszek commented Feb 23, 2022

robertoszek commented Feb 23, 2022 • edited

vserbu commented Feb 23, 2022

vserbu commented Feb 23, 2022 • edited

robertoszek commented Feb 23, 2022

vserbu commented Feb 23, 2022

robertoszek commented Feb 23, 2022 • edited

vserbu commented Feb 23, 2022 • edited

vserbu commented Feb 23, 2022

robertoszek commented Feb 23, 2022

robertoszek commented Feb 23, 2022

vserbu commented Feb 23, 2022

vserbu commented Feb 23, 2022 • edited

robertoszek commented Feb 24, 2022 • edited

vserbu commented Feb 24, 2022

robertoszek commented Feb 24, 2022

vserbu commented Feb 24, 2022

robertoszek commented Feb 24, 2022

nemobis commented Apr 3, 2022 • edited

robertoszek commented Apr 3, 2022

nemobis commented Apr 3, 2022 via email

nemobis commented Apr 10, 2022

robertoszek commented Apr 10, 2022 • edited

nemobis commented Apr 10, 2022 via email

nemobis commented Apr 22, 2022

robertoszek commented Apr 26, 2022

nemobis commented Jul 29, 2022

nemobis commented Jul 29, 2022

nemobis commented Nov 18, 2022

nemobis commented Nov 18, 2022

robertoszek commented Nov 18, 2022

nemobis commented Nov 18, 2022 via email

robertoszek commented Feb 23, 2022 •

edited

vserbu commented Feb 23, 2022 •

edited

robertoszek commented Feb 23, 2022 •

edited

vserbu commented Feb 23, 2022 •

edited

vserbu commented Feb 23, 2022 •

edited

robertoszek commented Feb 24, 2022 •

edited

nemobis commented Apr 3, 2022 •

edited

robertoszek commented Apr 10, 2022 •

edited