Skip to content
This repository has been archived by the owner on May 31, 2018. It is now read-only.

json request URI too large #209

Closed
sl1pkn07 opened this issue Jan 5, 2014 · 16 comments
Closed

json request URI too large #209

sl1pkn07 opened this issue Jan 5, 2014 · 16 comments
Labels
bug
Milestone
4.2.x - maintenance

Comments

@sl1pkn07
Copy link

sl1pkn07 commented Jan 5, 2014 

lexical error: invalid char in json text.
                                        <html>  <head><title>414 Reque
                     (right here) ------^

http://sl1pkn07.no-ip.com/paste/view/d58aaafa

greetings
@rmarquis
Copy link
Owner

rmarquis commented Jan 5, 2014

I cannot reproduce. It looks like the AUR Json interface when berksek for a moment, or that one of your packet is the reason of this issue.

If the error still occurs, please post a debug output (pastebin): "bash -x pacaur -Sua"

@sl1pkn07
Copy link
Author

sl1pkn07 commented Jan 5, 2014

bash -x pacaur -Sua log

http://sl1pkn07.no-ip.com/paste/view/9f005f26

rebuild cower, pacaur (or pacaur-git) no solve the problem

@rmarquis
Copy link
Owner

rmarquis commented Jan 5, 2014

There is a weird issue with the following packages, some of which aren't available in the AUR anymore:

x264-7mod
x264-devel
x264-jeeb
x264-kmod
x264-official
x264-tmod

Could you try to remove these packages, then run "pacaur -Sua" again, and see if the issue is solved? If yes, please then reinstall these packages with pacaur and see if the issue happens again.

@sl1pkn07
Copy link
Author

sl1pkn07 commented Jan 5, 2014

yes. these packages and much others are my personal packages (make by me and not upload to AUR).

I have not had problems with pacaur before, "orphan" packages (aka my personal packages) AND packages deleted from AUR listed OK before

EDIT: remove these packages not solve the problem, and I can not delete the other because I need

@rmarquis
Copy link
Owner

rmarquis commented Jan 5, 2014

Then please check your packaging, and most specifically the version number. According to the PKGBUILD guidelines, pkgver can contain only letters, numbers, periods and underscore. No hyphen nor plus sign. I suspect the latter to break the AUR RPC Json check.

@sl1pkn07
Copy link
Author

sl1pkn07 commented Jan 5, 2014

i found a installed package from AUR [1] with "+" in the version. I fix it but still fail

[1]https://aur.archlinux.org/packages/x265-hg/

NOTE: about x264 packages

the "-0.133.2334+704+51" is a name, not a version. the version is 0-1

example: http://sl1pkn07.no-ip.com/paste/view/cb21a3ff

@sl1pkn07
Copy link
Author

sl1pkn07 commented Jan 6, 2014

I found the problem

I edited the pacaur script and set exit in line 824

go to /tmp/pacaurtmp-sl1pkn07

the contents of json.raw is:

<html>
<head><title>414 Request-URI Too Large</title></head>
<body bgcolor="white">
<center><h1>414 Request-URI Too Large</h1></center>
<hr><center>nginx/1.4.4</center>
</body>
</html>

the uri is http://sl1pkn07.no-ip.com/paste/view/838ff632 (cut from bash -x pacaur -Sua)

@rmarquis
Copy link
Owner

rmarquis commented Jan 6, 2014

Yes, it might be. The maximum numbers of allowed packages per AUR RPC request is however currently set to 5000 (see $MAX_RPC_RESULTS in AUR sources), so your number is way lower than this. I don't see any settings about the max length of the URI.

@rmarquis
Copy link
Owner

rmarquis commented Jan 9, 2014

All right, so it seems the RPC AUR interface only allows GET request, so the max URI length is reached because of you very high number of AUR packages. The URI limit is 8190 characters by default.

I'm not sure how I'm going to solve this issue yet: cutting the whole Json request in half and trying to reconstruct a single json file after download, of simply falling back to the more slower info request method instead of multiinfo.

@rmarquis
Copy link
Owner

Note: I haven't pushed a fix for this yet in the new release, but I hope to fix this issue asap. I'm still unsure what would be the best way to handle it - any comment welcome!

@rmarquis rmarquis modified the milestones: 4.2.x - new features, 4.1.x - maintenance Mar 15, 2014
@rmarquis rmarquis modified the milestones: 4.1.x - maintenance, 4.2.x - new features Mar 23, 2014
@rmarquis rmarquis changed the title Json Request URI too large json request URI too large May 3, 2014
@rmarquis rmarquis mentioned this issue Jun 10, 2014
Closed
@wilzbach
Copy link

I am working currently on a way to split it in multiple parts and download them separately and them join them. Is there any ongoing other development for this issue that I should be aware of?

@rmarquis
Copy link
Owner

I started some work a few months ago but didn't really achieved a good result, so please carry on.
The ideal way of handling this issue would be to increase the limit server side, but I'm unsure about what upstream think about this idea (I didn't ask yet).

@wilzbach wilzbach mentioned this issue Jun 10, 2014
Closed
@wilzbach
Copy link

the workaround #238 does exactly what I described before. It splits the URL in multiple parts (e.g. 100) and then downloads them. The JSON joining is made with sed & awk (that is the ugly part).
One could probably increase the partSize to 250 or 300.

@rmarquis
Copy link
Owner

Should be implemented in 8f8350b in a somewhat simpler manner. I'm only "doubling" the URL length (more or less), so a request with about 700~800 packages should work. Hopefully, this is more than enough.

@greenify : Do you mind testing this change and see if this works for you?

@wilzbach
Copy link

sorry that it took me so long to reply. thank you! your changes work fine :)
However once someone has more than 600-800packages in the AUR you would need to have a loop.

@rmarquis
Copy link
Owner

Great. Yes, I'd need a loop but currently only 2 people so far seem to have reached the ~400 packages limit. Will certainly improve that part of the code if it becomes really necessary.

@coppolalab coppolalab mentioned this issue May 6, 2016
Closed
@AerysL AerysL mentioned this issue Aug 26, 2016
Closed
@rmarquis rmarquis mentioned this issue May 9, 2017
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
4.2.x - maintenance
Development

No branches or pull requests
3 participants
@rmarquis @sl1pkn07 @wilzbach