414 Request-URI Too Large

[kiraum]

Hello,

I'm having issues due a large request that I need to do using the nested recursive query below, this limit is an expect behavior? I can split the request, just to understand if there's a limit setted.

$ curl -sG https://peeringdb.com/api/net --data-urlencode field=asn,irr_as_set --data-urlencode depth=0 --data-urlencode asn__in=`curl -sG https://peeringdb.com/api/netixlan --data-urlencode ix_id=171 --data-urlencode depth=0 --data-urlencode field=asn | jq -c '[.data[].asn]' | sed 's/^.//;s/.$//'`

<title>414 Request-URI Too Large</title>

414 Request-URI Too Large

---

nginx

ASN list used above:
$ curl -sG https://peeringdb.com/api/netixlan --data-urlencode ix_id=171 --data-urlencode depth=0 --data-urlencode field=asn | jq -c '[.data[].asn]' | sed 's/^.//;s/.$//'
3549,31529,11271,28001,16685,6447,53070,42,3856,28166,1916,22356,30122,53116,53116,28220,53175,262792,25933,28165,16735,52925,28624,28338,28140,28299,28665,262787,14026,262680,262680,28187,28224,22431,28207,28207,52888,52888,28153,53145,28580,262561,28280,53216,28239,20406,20406,53114,28330,26162,26162,53066,20121,262760,28267,16397,16397,52613,262428,28591,27715,28589,28209,28209,28634,28634,11835,11835,262744,28250,262476,53053,263080,263080,28571,2688,1251,53166,262354,53165,262750,262750,262750,262750,53080,52720,262854,28666,25933,53187,28343,36408,28640,28241,15169,15169,8055,28311,22177,53184,28663,8075,52751,262375,16735,263444,53062,52937,262729,262729,52836,28300,28657,262659,28169,262757,262429,53171,8075,263471,28202,263442,27724,53140,61898,132203,132203,14868,262434,28663,28328,53037,53130,53142,262493,28636,262952,53111,53237,28146,53061,28171,262828,28309,262669,262704,262785,262785,262862,53008,53192,53178,28349,28349,28152,26162,26162,14571,262605,28271,28186,262891,28368,10733,262865,263055,28604,28604,28360,262737,262394,28306,26609,53001,52824,53222,53222,13335,262808,53221,262696,61924,262464,28132,262673,26615,28598,15133,263077,12654,20144,53072,263311,53115,53143,18881,263444,52892,17222,28233,7048,262336,262539,22177,52877,53088,14840,262848,263508,262372,7738,16509,16509,263042,52775,28294,28294,28292,262402,262632,14463,52579,28258,28168,28364,28364,53224,28124,28124,28250,263536,53122,52681,263544,262778,28370,262727,53242,263535,53102,53238,52550,262732,53102,263432,262293,28364,263297,262749,28191,264492,61581,262684,52908,262501,22689,28216,28329,53169,28283,52977,28359,262765,52736,263646,262375,28263,262847,263589,262902,53154,6057,53131,262700,262333,53230,264360,53112,14413,262720,19551,2906,52866,262288,264552,7162,263087,264101,262807,28670,36351,36236,262476,28135,262699,263584,52551,49544,262715,262667,263128,28654,52610,262761,262760,20940,53196,12989,263587,54113,54113,28219,263126,28287,264130,28352,52662,263558,61650,52858,28126,53158,28327,263668,262979,262664,262343,28226,28226,52689,262730,262743,28284,26615,4809,52930,22356,22356,262726,13414,28329,28329,28212,28212,21574,262454,264602,262790,28303,61928,264427,262868,28169,262669,262669,61942,28145,264268,52691,23106,262777,28590,52631,32934,52625,2635,52840,53013,263324,262612,28257,52863,262462,262346,28229,262620,53097,16397,16397,28328,262417,52877,263089,10310,10310,52697,28331,28331,52769,52769,53240,28667,28158,61832,262717,28125,262780,263555,263327,262444,263290,264005,22689,262427,262957,52940,30081,264531,61913,262457,28260,262413,28580,28668,263421,28307,14282,53018,263626,53246,6939,263047,28131,52683,262607,28279,28668,265187,28279,263269,1916,61673,53191,28576,28576,262784,264574,262481,263072,53078,28183,28198,28173,28173,27693,263049,52871,263453,52758,262949,53115,52765,262725,263353,53171,263314,262822,262317,263075,6507,6507,262748,265317,53233,28154,17222,28360,263661,264311,52911,52780,61785,263651,264009,262794,264116,263328,28667,32934,32934,262374,28587,262588,262908,263990,265306,263470,263152,28210,263394,53043,53218,263880,262517,263945,52691,262672,262672,263982,262469,265341,4230,262595,52320,52813,264244,263484,264016,262957,264144,262733,262733,262988,53075,52941,52604,52531,28577,264158,263044,28211,52989,18881,61584,262659,264543,263478,262589,53181,28605,262557,262802,52974,53059,264415,264415,262614,53152,52884,61580,53005,262822,262822,52686,263978,28323,52611,264575,2906,61568,28370,262983,52991,262527,53115,52770,53195,264083,52967,263070,264424,263433,262488,262720,263308,262427,61709,263567,25605,52532,53172,264114,262428,52971,61935,28267,263486,28305,53236,262968,52896,265065,264228,262585,262512,52965,262296,262296,28186,10733,53178,263990,264415,22085,28573,61871,263321,264525,262611,262865,264437,52848,262999,263154,263361,263329,262733,28130,28263,262388,264492,263276,266344,262426,262557,262532,53065,262408,52696,265303,262491,53115,264342,263415,262740,262650,264290,52901,264509,53137,52630,263655,28368,262489,266409,266468,264005,52878,54104,264397,263558,28642,266390,264913,28263,262509,53054,264911,264112,263110,14868,13357,28163,28178,52944,53162,53185,53197,262516,262568,262907,263460,53115,263269,53172,262952,262505,52669,264052,263665,263665,7048,263009,61890,28244,28154,28001,52752,52752,52752,61902,263112,265181,53013,53199,264332,263347,46489,46489,263542,52559,52602,263583,262268,262791,53067,265268,262700,53240,266112,52688,52688,265348,265348,262402,265144,61905,266413,263525,264367,266448,266498,263903,263903,262478,265260,8674,262691,262690,262690,264202,264202,263405,28580,264295,265437,52721,14551,262811,263311,28331,265300,265075,263077,262755,262755,263897,263877,61754,263569,262572,264170,28264,52981,23106,263336,265188,262688,264582,262893,264284,262503,264389,23148,28300,264527,262739,265241,265464,61864,266135,262338,262979,53018,262365,264983,262466,61689,265347,263356,28368,262355,262355,262548,264119,52596,263470,264580,264861,264069,53202,262644,28328,263044,263044,22548,53139,28368,263625,263270,265185,262311,266201,263042,262462,264071,264223,262819,20940,28666,266368,53065,28656,264564,263265,262730,262730,263964,265027,266210,264409,263297,16685,262409,263905,52822,266374,266406,264597,265438,52923,263143,61586,53170,265950,53138,53240,23074,264233,53180,52720,264486,262354,265141,264959,262731,262731,265308,264262,264022,262643,262685,28639,263124,263912,264425,28657,28189,28279,266131,32934,61951,266475,264116,264332,264332,265061,28598,262993,262834,28668,61805,263084,263374,33182,52654,52654,264031,263120,61704,52601,265498,264230,262734,264186,27471,263042,263283,264479,266437,264893,266167,52650,262838,262838,265036,267460,44444,53113,262766,52650,52721,266177,265496,265949,265949,264574,265377,263445,264382,264382,28166,262766,262588,264062,11844,265933,263595,262809,262809,265184,53209,265436,263626,266523,266365,266365,262725,262669,262469,11338,265030,264415,262683,52911,265429,263086,263019,265021,262729,61570,266387,23201,262463,262463,266525,23201,53045,266137,265389,262546,61582,263142,262343,265238,262848,263151,28143,28143,53008,263920,263411,28265,263918,264480,53145,28624,28277,264877,262610,52544,262549,61888,266165,264238,264381,263073,28229,262302,267586,53180,263251,53180,266167,264574,52666,52937,266211,266534,265478,262448,262745,28649,28649,52531,52712,265980,262444,263401,263401,266995,265427,263386,23105,23105,15256,264948,53094,264331,52887,27715,53106,262669,52547,262632,61943,263968,265937,61822,263384,264409,264130,266302,53132,61873,28638,28638,267522,28139,53095,263328,52686,53231,264460,28638,264450,267613,262336,28139,53147,264367,52871,266445,28280,199524,266420,262502,28263,28309,36692,266518,22689,267673,61659,267452,264359,28171,262775,16734,263985,266977,263535,263072,262983,263434,262671,28598,266031,264189,52873,264293,266177,266156,53115,264002,263548,53196,52726,262323,266468,61587,53087,266574,264426,266523,262761,53127,264344,262862,265016,264096,262729,263003,262676,28669,266633,265356,262516,262946,263675,264590,262818,266039,52542,262518,262518,52952,266121,262681,14204,263572,262807,53107,263043,264992,264992,266260,265952,267196,263649,264020,264555,16509,16509,264461,266136,52539,265950,265950,264380,264415,262865,264453,266926,262468,267428,262587,267276,22381,264479,263461,266951,267530,267530,267124,266341,263591,262770,28156,265147,263647,53152,265999,61730,263963,263032,265158,263855,61586,61586,53048,52527,262901,37468,262307,28277,267075,28158,28158,264127,264127,262662,267294,267544,263541,262711,61910,266435,264023,263661,264023,262424,266028,28610,6447,265886,262605,52932,264491,266623,266300,52932,267443,28580,268225,263009,6057,52936,263362,266925,266925,265936,61832,263053,52977,268222,263053,263053,267264,266652,264863,264518,28145,262468,42473,53086,264152,28657,264949,265330,265959,263531,263531,263531,267170,262353,264339,264339,264339,265347,267085,262469,61588,11338,53156,267088,267088,262977,28580,262605,267124,266112,265912,262287,28252,265019,265209,52664,262594,266993,53050,264598,28125,267121,268221,262589,61749,262688,52661,262688,267280,263018,263606,28263,28130,262617,262403,263421,265030,265206,265207,264518,264191

Without the recursive request same results:
$ curl -sG https://peeringdb.com/api/net --data-urlencode field=asn,irr_as_set --data-urlencode depth=0 --data-urlencode asn__in=3549,31529,11271,28001,16685,6447,53070,42,3856,28166,1916,22356,30122,53116,53116,28220,53175,262792,25933,28165,16735,52925,28624,28338,28140,28299,28665,262787,14026,262680,262680,28187,28224,22431,28207,28207,52888,52888,28153,53145,28580,262561,28280,53216,28239,20406,20406,53114,28330,26162,26162,53066,20121,262760,28267,16397,16397,52613,262428,28591,27715,28589,28209,28209,28634,28634,11835,11835,262744,28250,262476,53053,263080,263080,28571,2688,1251,53166,262354,53165,262750,262750,262750,262750,53080,52720,262854,28666,25933,53187,28343,36408,28640,28241,15169,15169,8055,28311,22177,53184,28663,8075,52751,262375,16735,263444,53062,52937,262729,262729,52836,28300,28657,262659,28169,262757,262429,53171,8075,263471,28202,263442,27724,53140,61898,132203,132203,14868,262434,28663,28328,53037,53130,53142,262493,28636,262952,53111,53237,28146,53061,28171,262828,28309,262669,262704,262785,262785,262862,53008,53192,53178,28349,28349,28152,26162,26162,14571,262605,28271,28186,262891,28368,10733,262865,263055,28604,28604,28360,262737,262394,28306,26609,53001,52824,53222,53222,13335,262808,53221,262696,61924,262464,28132,262673,26615,28598,15133,263077,12654,20144,53072,263311,53115,53143,18881,263444,52892,17222,28233,7048,262336,262539,22177,52877,53088,14840,262848,263508,262372,7738,16509,16509,263042,52775,28294,28294,28292,262402,262632,14463,52579,28258,28168,28364,28364,53224,28124,28124,28250,263536,53122,52681,263544,262778,28370,262727,53242,263535,53102,53238,52550,262732,53102,263432,262293,28364,263297,262749,28191,264492,61581,262684,52908,262501,22689,28216,28329,53169,28283,52977,28359,262765,52736,263646,262375,28263,262847,263589,262902,53154,6057,53131,262700,262333,53230,264360,53112,14413,262720,19551,2906,52866,262288,264552,7162,263087,264101,262807,28670,36351,36236,262476,28135,262699,263584,52551,49544,262715,262667,263128,28654,52610,262761,262760,20940,53196,12989,263587,54113,54113,28219,263126,28287,264130,28352,52662,263558,61650,52858,28126,53158,28327,263668,262979,262664,262343,28226,28226,52689,262730,262743,28284,26615,4809,52930,22356,22356,262726,13414,28329,28329,28212,28212,21574,262454,264602,262790,28303,61928,264427,262868,28169,262669,262669,61942,28145,264268,52691,23106,262777,28590,52631,32934,52625,2635,52840,53013,263324,262612,28257,52863,262462,262346,28229,262620,53097,16397,16397,28328,262417,52877,263089,10310,10310,52697,28331,28331,52769,52769,53240,28667,28158,61832,262717,28125,262780,263555,263327,262444,263290,264005,22689,262427,262957,52940,30081,264531,61913,262457,28260,262413,28580,28668,263421,28307,14282,53018,263626,53246,6939,263047,28131,52683,262607,28279,28668,265187,28279,263269,1916,61673,53191,28576,28576,262784,264574,262481,263072,53078,28183,28198,28173,28173,27693,263049,52871,263453,52758,262949,53115,52765,262725,263353,53171,263314,262822,262317,263075,6507,6507,262748,265317,53233,28154,17222,28360,263661,264311,52911,52780,61785,263651,264009,262794,264116,263328,28667,32934,32934,262374,28587,262588,262908,263990,265306,263470,263152,28210,263394,53043,53218,263880,262517,263945,52691,262672,262672,263982,262469,265341,4230,262595,52320,52813,264244,263484,264016,262957,264144,262733,262733,262988,53075,52941,52604,52531,28577,264158,263044,28211,52989,18881,61584,262659,264543,263478,262589,53181,28605,262557,262802,52974,53059,264415,264415,262614,53152,52884,61580,53005,262822,262822,52686,263978,28323,52611,264575,2906,61568,28370,262983,52991,262527,53115,52770,53195,264083,52967,263070,264424,263433,262488,262720,263308,262427,61709,263567,25605,52532,53172,264114,262428,52971,61935,28267,263486,28305,53236,262968,52896,265065,264228,262585,262512,52965,262296,262296,28186,10733,53178,263990,264415,22085,28573,61871,263321,264525,262611,262865,264437,52848,262999,263154,263361,263329,262733,28130,28263,262388,264492,263276,266344,262426,262557,262532,53065,262408,52696,265303,262491,53115,264342,263415,262740,262650,264290,52901,264509,53137,52630,263655,28368,262489,266409,266468,264005,52878,54104,264397,263558,28642,266390,264913,28263,262509,53054,264911,264112,263110,14868,13357,28163,28178,52944,53162,53185,53197,262516,262568,262907,263460,53115,263269,53172,262952,262505,52669,264052,263665,263665,7048,263009,61890,28244,28154,28001,52752,52752,52752,61902,263112,265181,53013,53199,264332,263347,46489,46489,263542,52559,52602,263583,262268,262791,53067,265268,262700,53240,266112,52688,52688,265348,265348,262402,265144,61905,266413,263525,264367,266448,266498,263903,263903,262478,265260,8674,262691,262690,262690,264202,264202,263405,28580,264295,265437,52721,14551,262811,263311,28331,265300,265075,263077,262755,262755,263897,263877,61754,263569,262572,264170,28264,52981,23106,263336,265188,262688,264582,262893,264284,262503,264389,23148,28300,264527,262739,265241,265464,61864,266135,262338,262979,53018,262365,264983,262466,61689,265347,263356,28368,262355,262355,262548,264119,52596,263470,264580,264861,264069,53202,262644,28328,263044,263044,22548,53139,28368,263625,263270,265185,262311,266201,263042,262462,264071,264223,262819,20940,28666,266368,53065,28656,264564,263265,262730,262730,263964,265027,266210,264409,263297,16685,262409,263905,52822,266374,266406,264597,265438,52923,263143,61586,53170,265950,53138,53240,23074,264233,53180,52720,264486,262354,265141,264959,262731,262731,265308,264262,264022,262643,262685,28639,263124,263912,264425,28657,28189,28279,266131,32934,61951,266475,264116,264332,264332,265061,28598,262993,262834,28668,61805,263084,263374,33182,52654,52654,264031,263120,61704,52601,265498,264230,262734,264186,27471,263042,263283,264479,266437,264893,266167,52650,262838,262838,265036,267460,44444,53113,262766,52650,52721,266177,265496,265949,265949,264574,265377,263445,264382,264382,28166,262766,262588,264062,11844,265933,263595,262809,262809,265184,53209,265436,263626,266523,266365,266365,262725,262669,262469,11338,265030,264415,262683,52911,265429,263086,263019,265021,262729,61570,266387,23201,262463,262463,266525,23201,53045,266137,265389,262546,61582,263142,262343,265238,262848,263151,28143,28143,53008,263920,263411,28265,263918,264480,53145,28624,28277,264877,262610,52544,262549,61888,266165,264238,264381,263073,28229,262302,267586,53180,263251,53180,266167,264574,52666,52937,266211,266534,265478,262448,262745,28649,28649,52531,52712,265980,262444,263401,263401,266995,265427,263386,23105,23105,15256,264948,53094,264331,52887,27715,53106,262669,52547,262632,61943,263968,265937,61822,263384,264409,264130,266302,53132,61873,28638,28638,267522,28139,53095,263328,52686,53231,264460,28638,264450,267613,262336,28139,53147,264367,52871,266445,28280,199524,266420,262502,28263,28309,36692,266518,22689,267673,61659,267452,264359,28171,262775,16734,263985,266977,263535,263072,262983,263434,262671,28598,266031,264189,52873,264293,266177,266156,53115,264002,263548,53196,52726,262323,266468,61587,53087,266574,264426,266523,262761,53127,264344,262862,265016,264096,262729,263003,262676,28669,266633,265356,262516,262946,263675,264590,262818,266039,52542,262518,262518,52952,266121,262681,14204,263572,262807,53107,263043,264992,264992,266260,265952,267196,263649,264020,264555,16509,16509,264461,266136,52539,265950,265950,264380,264415,262865,264453,266926,262468,267428,262587,267276,22381,264479,263461,266951,267530,267530,267124,266341,263591,262770,28156,265147,263647,53152,265999,61730,263963,263032,265158,263855,61586,61586,53048,52527,262901,37468,262307,28277,267075,28158,28158,264127,264127,262662,267294,267544,263541,262711,61910,266435,264023,263661,264023,262424,266028,28610,6447,265886,262605,52932,264491,266623,266300,52932,267443,28580,268225,263009,6057,52936,263362,266925,266925,265936,61832,263053,52977,268222,263053,263053,267264,266652,264863,264518,28145,262468,42473,53086,264152,28657,264949,265330,265959,263531,263531,263531,267170,262353,264339,264339,264339,265347,267085,262469,61588,11338,53156,267088,267088,262977,28580,262605,267124,266112,265912,262287,28252,265019,265209,52664,262594,266993,53050,264598,28125,267121,268221,262589,61749,262688,52661,262688,267280,263018,263606,28263,28130,262617,262403,263421,265030,265206,265207,264518,264191

<title>414 Request-URI Too Large</title>

414 Request-URI Too Large

---

nginx

Thanks in advance.

[arnoldnipper]

@vegu / @grizz we already had this issue ... please increase the buffer size to a reasonable value. The earlier the better

[vegu]

@kiraum @arnoldnipper i have forwarded this issue to the ops team as it is a server configuration issue and a new reasonable limit needs to be determined - the buffer size is still the same as from the time we increased it to fix the issue then, so it's simply people hitting the new limit

[kiraum]

Thanks @arnoldnipper and @vegu, please if possible let me know when the limit is increased.

[arnoldnipper]

Another user reported about "502 Bad gateway" when doing https://peeringdb.com/api/net?ix=18&depth=2 ... maybe that's related

[vegu]

@arnoldnipper Different issue - appears it's taking too long to serve the response and the change from #252 is killing the request - i will email ops to increase the timeout limit

[arnoldnipper]

@vegu ... this explains the behaviour user is reporting. He says that's sometime happening, sometimes not. Thanks!

[ccaputo]

Harakiri has been increased from 25 to 60 seconds. Arnold, please let the user know the issue should be resolved. Thanks!

[arnoldnipper]

Thanks, @ccaputo ... what about $subject itself?!?

[ccaputo]

The production site now handles the above query and queries up to approximately 50% longer. The actual max query length is hard to quantify due to HTML escaping, but with HTML escaping figure about 16k. In a test, prior to HTML escaping, I was able to do a query over 12k. Some details for future reference should it come time we need to increase further:

nginx.conf: large_client_header_buffers 4 16k;

• Synced to uwsgi's "buffer-size=65535", which is the limit of uwsgi protocol also since query string is essentially sent twice, once as QUERY_STRING and once in REQUEST_URI, and because of other variables that get sent, we divide uwsgi's limit by 2. Until an nginx bug is fixed (patch submitted 2018-08-22), in circumstances when nginx overflows a uint16 and tries to send a length exceeding the uwsgi protocol, uwsgi will potentially reply with a bogus response. Thus dividing by another 2 to try to avoid that. Once nginx's ngx_http_uwsgi_create_request len check() is fixed, we can explore raising this to 32k.

[ccaputo]

The nginx fix mentioned above is: https://hg.nginx.org/nginx/rev/f6e7831a17d4

I expect it will be in upcoming nginx-1.15.4. Once that or better is on the beta and production front end servers, we can adjust nginx.conf to have 32k header buffers.

[ccaputo]

Confirmed nginx fix is in nginx-1.15.4. Prod1 & 2 are now running nginx-1.16.1 while beta1 & 2 are still running nginx-1.12.2. If there is interest in larger query handling, I am comfortable with production changing to 32k buffers per my Aug 30, 2018 comment above. Ie.: this could be done:

nginx.conf: large_client_header_buffers 4 32k;

[grizz]

@peeringdb/oc Let's do 32k for next release?

[arnoldnipper]

+1

[ccaputo]

@grizz is this something to configure inside of a container or does nginx run outside of a container?

[ynbrthr]

is this still an issue that needs to be addressed? If so, +1

[ccaputo]

On beta I tested a change in nginx.conf from large_client_header_buffers 4 16k; to large_client_header_buffers 4 32k; and found that I was only able to increase the request header size maximum, in query tests, by 12 bytes. My ASCII request byte count test was able to go from 12,505 bytes to 12,517 bytes. Thinking we might be hitting a limit with AWS's Application Load Balancer, I found that we are! From https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html:

Thus I am closing this and leaving nginx.conf at large_client_header_buffers 4 16k; which is sufficient for the query which prompted this issue in the first place. I don't think the extra 12 bytes is worth the regression testing and other bugs we might expose in nginx by the change, by running something as non-standard as this.