Failed to acquire write lock for id: -333

[renaldas]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

11.3.1

Bug summary

When trying to save node getting this error, it happens on random nodes, sometimes you need to hit 3 times to publish the page.

Site runs on Azure with Azure SQL, all configuration set as per recommendations for Azure and also just tried to save and publish page locally, same thing happens.

Specifics

No response

Steps to reproduce

Expected result / actual result

No response

[github-actions]

Hi there @renaldas!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[Zeegaan]

@renaldas What configuration are you running with? I know you said as per recommendations, but are those from the docs or ? 🤔 (Regardless, I'd love a link 👍)

[renaldas]

@Zeegaan

Config based on: https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/azure-web-apps

I also increased: DistributedLockingReadLockDefaultTimeout and DistributedLockingWriteLockDefaultTimeout to max

"CMS": {
      "ModelsBuilder": {
        "ModelsMode": "Nothing"
      },
      "Global": {
        "UseHttps": true,
        "MainDomLock": "FileSystemMainDomLock",
        "DistributedLockingReadLockDefaultTimeout": "00:02:00",
        "DistributedLockingWriteLockDefaultTimeout": "00:00:20"
      },
      "Hosting": {
        "LocalTempStorageLocation": "EnvironmentTemp"
      },
      "RuntimeMinification": {
        "UseInMemoryCache": true,
        "CacheBuster": "Version",
        "Version": "20230504"
      },
      "Examine": {
        "LuceneDirectoryFactory": "SyncedTempFileSystemDirectoryFactory"
      },
      "Content": {
        "ContentVersionCleanupPolicy": {
          "EnableCleanup": false
        },
      }    
    }

[tnielsenskybruddk]

I experience the same problem in a 10.4.2 solution. Also running on Azure SQL.

My configuration looks like this:

    "CMS": {
      "Runtime": {
        "MaxRequestLength": 40000
      },
      "Global": {
        "SanitizeTinyMce": true,
        "ReservedUrls": "~/sitemap.xml",
        "ReservedPaths": "~/api/,~/swagger",
        "UseHttps": true,
        "TimeOut": "02:00:00",
        "Smtp": {
          "From": "",
          "Host": "",
          "Port": 25
        }
      },
      "RuntimeMinification": {
        "UseInMemoryCache": false,
        "CacheBuster": "Version",
        "Version": "49"
      },
      "Content": {
        "ContentVersionCleanupPolicy": {
          "EnableCleanup": true
        }
      },
      "ModelsBuilder": {
        "ModelsMode": "Nothing",
        "ModelsDirectory": "~/../code/Models/Umbraco",
        "ModelsNamespace": "Code.Models.Umbraco",
        "UseDirectories": true,
        "AcceptUnsafeModelsDirectory": true
      },
      "RichTextEditor": {
        "CustomConfig": {
          "advlist_bullet_styles": "default",
          "advlist_number_styles": "standard,upper-alpha"
        }
      },
      "Security": {
        "AllowEditInvariantFromNonDefault": "true"
      }
    },

[Zeegaan]

@tnielsenskybruddk is this something happening to you frequently? And how frequent?🤔

[tnielsenskybruddk]

@Zeegaan That is very diffent. Last week it came once in a while. At the moment it comes all the time.

I first experienced the problem on April 27, 2023.

[bjarnef]

@Zeegaan I am frequently seeing this on an Umbraco v10.4.0 project. Often when saving content types.

I often get this error when working locally.

Waiting a few minutes the document type is saved without errors.

[bjarnef]

@Zeegaan we are also seeing this error sometimes when deploying to Development environment where deployment extraction fails and clear signatures + new extraction with Deploy dashboard isn't sufficient.

A recycle of the environment seems to fix it. I guess it is a similar issue in cache refreshing when Deploy tries to save the conntent type.

[anthonycodedynamo]

I'm seeing this on a local dev environment with a local SQL Server when running a tight loop to unpublish 1000s of nodes. I suspect it has something to do with GetRemainingTimeoutSeconds and the logged in user's remaining session time.

I found after rebooting IIS and keeping the session a live I managed to side-step the error. I can't find documentation on Umbraco 10+ on how to extend the session time.

Also, I'd be interested on more details of DistributedLockingReadLockDefaultTimeout and DistributedLockingWriteLockDefaultTimeout, presuming these are database row lock timeouts.

Not trying to hijack the issue here... please ignore if this isn't relevant to the issue at hand.

[russellshome]

It seems that this query is closely associated with this issue and at times takes an inordinately long time to complete

exec sp_executesql N'UPDATE umbracoLock WITH (REPEATABLEREAD) SET value = (CASE WHEN (value=1) THEN -1 ELSE 1 END) WHERE id=@0',N'@0 int',@0=-333

[russellshome]

I am able to get the problem to happen every time if I publish the home page and then publish another page.
I ran SQL Profiler and these queries while umbraco was trying to save the other page:

select cmd,* from sys.sysprocesses
where blocked > 0

SELECT * FROM sys.dm_tran_locks
  WHERE resource_database_id = DB_ID()
  AND resource_associated_entity_id = OBJECT_ID(N'dbo.umbracoLock');

That revealed a query locking the umbracoLock table that took over 25 seconds to complete - see below.

I'm not sure yet what is running this query or why but it seems like it tries to get every property of every page. In my case over 15000 rows

exec sp_executesql N'SELECT [umbracoPropertyData].[id] AS [Id], [umbracoPropertyData].[versionId] AS [VersionId], [umbracoPropertyData].[propertyTypeId] AS [PropertyTypeId], [umbracoPropertyData].[languageId] AS [LanguageId], [umbracoPropertyData].[segment] AS [Segment], [umbracoPropertyData].[intValue] AS [IntegerValue], [umbracoPropertyData].[decimalValue] AS [DecimalValue], [umbracoPropertyData].[dateValue] AS [DateValue], [umbracoPropertyData].[varcharValue] AS [VarcharValue], [umbracoPropertyData].[textValue] AS [TextValue]
FROM [umbracoPropertyData]
WHERE ([umbracoPropertyData].[versionId] IN (@0,@1,@2,@3,@4,@5,@6,@7,@8,@9,@10,@11,@12,@13,@14,@15,@16,@17,@18,@19,@20,@21,@22,@23,@24,@25,@26,@27,@28,@29,@30,@31,@32,@33,@34,@35,@36,@37,@38,@39,@40,@41,@42,@43,@44,@45,@46,@47,@48,@49,@50,@51,@52,@53,@54,@55,@56,@57,@58,@59,@60,@61,@62,@63,@64,@65,@66,@67,@68,@69,@70,@71,@72,@73,@74,@75,@76,@77,@78,@79,@80,@81,@82,@83,@84,@85,@86,@87,@88,@89,@90,@91,@92,@93,@94,@95,@96,@97,@98,@99,@100,@101,@102,@103,@104,@105,@106,@107,@108,@109,@110,@111,@112,@113,@114,@115,@116,@117,@118,@119,@120,@121,@122,@123,@124,@125,@126,@127,@128,@129,@130,@131,@132,@133,@134,@135,@136,@137,@138,@139,@140,@141,@142,@143,@144,@145,@146,@147,@148,@149,@150,@151,@152,@153,@154,@155,@156,@157,@158,@159,@160,@161,@162,@163,@164,@165,@166,@167,@168,@169,@170,@171,@172,@173,@174,@175,@176,@177,@178,@179,@180,@181,@182,@183,@184,@185,@186,@187,@188,@189,@190,@191,@192,@193,@194,@195,@196,@197,@198,@199,@200,@201,@202,@203,@204,@205,@206,@207,@208,@209,@210,@211,@212,@213,@214,@215,@216,@217,@218,@219,@220,@221,@222,@223,@224,@225,@226,@227,@228,@229,@230,@231,@232,@233,@234,@235,@236,@237,@238,@239,@240,@241,@242,@243,@244,@245,@246,@247,@248,@249,@250,@251,@252,@253,@254,@255,@256,@257,@258,@259,@260,@261,@262,@263,@264,@265,@266,@267,@268,@269,@270,@271,@272,@273,@274,@275,@276,@277,@278,@279,@280,@281,@282,@283,@284,@285,@286,@287,@288,@289,@290,@291,@292,@293,@294,@295,@296,@297,@298,@299,@300,@301,@302,@303,@304,@305,@306,@307,@308,@309,@310,@311,@312,@313,@314,@315,@316,@317,@318,@319,@320,@321,@322,@323,@324,@325,@326,@327,@328,@329,@330,@331,@332,@333,@334,@335,@336,@337,@338,@339,@340,@341,@342,@343,@344,@345,@346,@347,@348,@349,@350,@351,@352,@353,@354,@355,@356,@357,@358,@359,@360,@361,@362,@363,@364,@365,@366,@367,@368,@369,@370,@371,@372,@373,@374,@375,@376,@377,@378,@379,@380,@381,@382,@383,@384,@385,@386,@387,@388,@389,@390,@391,@392,@393,@394,@395,@396,@397,@398,@399,@400,@401,@402,@403,@404,@405,@406,@407,@408,@409,@410,@411,@412,@413,@414,@415,@416,@417,@418,@419,@420,@421,@422,@423,@424,@425,@426,@427,@428,@429,@430,@431,@432,@433,@434,@435,@436,@437,@438,@439,@440,@441,@442,@443,@444,@445,@446,@447,@448,@449,@450,@451,@452,@453,@454,@455,@456,@457,@458,@459,@460,@461,@462,@463,@464,@465,@466,@467,@468,@469,@470,@471,@472,@473,@474,@475,@476,@477,@478,@479,@480,@481,@482,@483,@484,@485,@486,@487,@488,@489,@490,@491,@492,@493,@494,@495,@496,@497,@498,@499,@500,@501,@502,@503,@504,@505,@506,@507,@508,@509,@510,@511,@512,@513,@514,@515,@516,@517,@518,@519,@520,@521,@522,@523,@524,@525,@526,@527,@528,@529,@530,@531,@532,@533,@534,@535,@536,@537,@538,@539,@540,@541,@542,@543,@544,@545,@546,@547,@548,@549,@550,@551,@552,@553,@554,@555,@556,@557,@558,@559,@560,@561,@562,@563,@564,@565,@566,@567,@568,@569,@570,@571,@572,@573,@574,@575,@576,@577,@578,@579,@580,@581,@582,@583,@584,@585,@586,@587,@588,@589,@590,@591,@592,@593,@594,@595,@596,@597,@598,@599,@600,@601,@602,@603,@604,@605,@606,@607,@608,@609,@610,@611,@612,@613,@614,@615,@616,@617,@618,@619,@620,@621,@622,@623,@624,@625,@626,@627,@628,@629,@630,@631,@632,@633,@634,@635,@636,@637,@638,@639,@640,@641,@642,@643,@644,@645,@646,@647,@648,@649,@650,@651,@652,@653,@654,@655,@656,@657,@658,@659,@660,@661,@662,@663,@664,@665,@666,@667,@668,@669,@670,@671,@672,@673,@674,@675,@676,@677,@678,@679,@680,@681,@682,@683,@684,@685,@686,@687,@688,@689,@690,@691,@692,@693,@694,@695,@696,@697,@698,@699,@700,@701,@702,@703,@704,@705,@706,@707,@708,@709,@710,@711,@712,@713,@714,@715,@716,@717,@718,@719,@720,@721,@722,@723,@724,@725,@726,@727,@728,@729,@730,@731,@732,@733,@734,@735,@736,@737,@738,@739,@740,@741,@742,@743,@744,@745,@746,@747,@748,@749,@750,@751,@752,@753,@754,@755,@756,@757,@758,@759,@760,@761,@762,@763,@764,@765,@766,@767,@768,@769,@770,@771,@772,@773,@774,@775,@776,@777,@778,@779,@780,@781,@782,@783,@784,@785,@786,@787,@788,@789,@790,@791,@792,@793,@794,@795,@796,@797,@798,@799,@800,@801,@802,@803,@804,@805,@806,@807,@808,@809,@810,@811,@812,@813,@814,@815,@816,@817,@818,@819,@820,@821,@822,@823,@824,@825,@826,@827,@828,@829,@830,@831,@832,@833,@834,@835,@836,@837,@838,@839,@840,@841,@842,@843,@844,@845,@846,@847,@848,@849,@850,@851,@852,@853,@854,@855,@856,@857,@858,@859,@860,@861,@862,@863,@864,@865,@866,@867,@868,@869,@870,@871,@872,@873,@874,@875,@876,@877,@878,@879,@880,@881,@882,@883,@884,@885,@886,@887,@888,@889,@890,@891,@892,@893,@894,@895,@896,@897,@898,@899,@900,@901,@902,@903,@904,@905,@906,@907,@908,@909,@910,@911,@912,@913,@914,@915,@916,@917,@918,@919,@920,@921,@922,@923,@924,@925,@926,@927,@928,@929,@930,@931,@932,@933,@934,@935,@936,@937,@938,@939,@940,@941,@942,@943,@944,@945,@946,@947,@948,@949,@950,@951,@952,@953,@954,@955,@956,@957,@958,@959,@960,@961,@962,@963,@964,@965,@966,@967,@968,@969,@970,@971,@972,@973,@974,@975,@976,@977,@978,@979,@980,@981,@982,@983,@984,@985,@986,@987,@988,@989,@990,@991,@992,@993,@994,@995,@996,@997,@998,@999,@1000,@1001,@1002,@1003,@1004,@1005,@1006,@1007,@1008,@1009,@1010,@1011,@1012,@1013,@1014,@1015,@1016,@1017,@1018,@1019,@1020,@1021,@1022,@1023,@1024,@1025,@1026,@1027,@1028,@1029,@1030,@1031,@1032,@1033,@1034,@1035,@1036,@1037,@1038,@1039,@1040,@1041,@1042,@1043,@1044,@1045,@1046,@1047,@1048,@1049,@1050,@1051,@1052,@1053,@1054,@1055,@1056,@1057,@1058,@1059,@1060,@1061,@1062,@1063,@1064,@1065,@1066,@1067,@1068,@1069,@1070,@1071,@1072,@1073,@1074,@1075,@1076,@1077,@1078,@1079,@1080,@1081,@1082,@1083,@1084,@1085,@1086,@1087,@1088,@1089,@1090,@1091,@1092,@1093,@1094,@1095,@1096,@1097,@1098,@1099,@1100,@1101,@1102,@1103,@1104,@1105,@1106,@1107,@1108,@1109,@1110,@1111,@1112,@1113,@1114,@1115,@1116,@1117,@1118,@1119,@1120,@1121,@1122,@1123,@1124,@1125,@1126,@1127,@1128,@1129,@1130,@1131,@1132,@1133,@1134,@1135,@1136,@1137,@1138,@1139,@1140,@1141,@1142,@1143,@1144,@1145,@1146,@1147,@1148,@1149,@1150,@1151,@1152,@1153,@1154,@1155,@1156,@1157,@1158,@1159,@1160,@1161,@1162,@1163,@1164,@1165,@1166,@1167,@1168,@1169,@1170,@1171,@1172,@1173,@1174,@1175,@1176,@1177,@1178,@1179,@1180,@1181,@1182,@1183,@1184,@1185,@1186,@1187,@1188,@1189,@1190,@1191,@1192,@1193,@1194,@1195,@1196,@1197,@1198,@1199,@1200,@1201,@1202,@1203,@1204,@1205,@1206,@1207,@1208,@1209,@1210,@1211,@1212,@1213,@1214,@1215,@1216,@1217,@1218,@1219,@1220,@1221,@1222,@1223,@1224,@1225,@1226,@1227,@1228,@1229,@1230,@1231,@1232,@1233,@1234,@1235,@1236,@1237,@1238,@1239,@1240,@1241,@1242,@1243,@1244,@1245,@1246,@1247,@1248,@1249,@1250,@1251,@1252,@1253,@1254,@1255,@1256,@1257,@1258,@1259,@1260,@1261,@1262,@1263,@1264,@1265,@1266,@1267,@1268,@1269,@1270,@1271,@1272,@1273,@1274,@1275,@1276,@1277,@1278,@1279,@1280,@1281,@1282,@1283,@1284,@1285,@1286,@1287,@1288,@1289,@1290,@1291,@1292,@1293,@1294,@1295,@1296,@1297,@1298,@1299,@1300,@1301,@1302,@1303,@1304,@1305,@1306,@1307,@1308,@1309,@1310,@1311,@1312,@1313,@1314,@1315,@1316,@1317,@1318,@1319,@1320,@1321,@1322,@1323,@1324,@1325,@1326,@1327,@1328,@1329,@1330,@1331,@1332,@1333,@1334,@1335,@1336,@1337,@1338,@1339,@1340,@1341,@1342,@1343,@1344,@1345,@1346,@1347,@1348,@1349,@1350,@1351,@1352,@1353,@1354,@1355,@1356,@1357,@1358,@1359,@1360,@1361,@1362,@1363,@1364,@1365,@1366,@1367,@1368,@1369,@1370,@1371,@1372,@1373,@1374,@1375,@1376,@1377,@1378,@1379,@1380,@1381,@1382,@1383,@1384,@1385,@1386,@1387,@1388,@1389,@1390,@1391,@1392,@1393,@1394,@1395,@1396,@1397,@1398,@1399,@1400,@1401,@1402,@1403,@1404,@1405,@1406,@1407,@1408,@1409,@1410,@1411,@1412,@1413,@1414,@1415,@1416,@1417,@1418,@1419,@1420,@1421,@1422,@1423,@1424,@1425,@1426,@1427,@1428,@1429,@1430,@1431,@1432,@1433,@1434,@1435,@1436,@1437,@1438,@1439,@1440,@1441,@1442,@1443,@1444,@1445,@1446,@1447,@1448,@1449,@1450,@1451,@1452,@1453,@1454,@1455,@1456,@1457,@1458,@1459,@1460,@1461,@1462,@1463,@1464,@1465,@1466,@1467,@1468,@1469,@1470,@1471,@1472,@1473,@1474,@1475,@1476,@1477,@1478,@1479,@1480,@1481,@1482,@1483,@1484,@1485,@1486,@1487,@1488,@1489,@1490,@1491,@1492,@1493,@1494,@1495,@1496,@1497,@1498,@1499,@1500,@1501,@1502,@1503,@1504,@1505,@1506,@1507,@1508,@1509,@1510,@1511,@1512,@1513,@1514,@1515,@1516,@1517,@1518,@1519,@1520,@1521,@1522,@1523,@1524,@1525,@1526,@1527,@1528,@1529,@1530,@1531,@1532,@1533,@1534,@1535,@1536,@1537,@1538,@1539,@1540,@1541,@1542,@1543,@1544,@1545,@1546,@1547,@1548,@1549,@1550,@1551,@1552,@1553,@1554,@1555,@1556,@1557,@1558,@1559,@1560,@1561,@1562,@1563,@1564,@1565,@1566,@1567,@1568,@1569,@1570,@1571,@1572,@1573,@1574,@1575,@1576,@1577,@1578,@1579,@1580,@1581,@1582,@1583,@1584,@1585,@1586,@1587,@1588,@1589,@1590,@1591,@1592,@1593,@1594,@1595,@1596,@1597,@1598,@1599,@1600,@1601,@1602,@1603,@1604,@1605,@1606,@1607,@1608,@1609,@1610,@1611,@1612,@1613,@1614,@1615,@1616,@1617,@1618,@1619,@1620,@1621,@1622,@1623,@1624,@1625,@1626,@1627,@1628,@1629,@1630,@1631,@1632,@1633,@1634,@1635,@1636,@1637,@1638,@1639,@1640,@1641,@1642,@1643,@1644,@1645,@1646,@1647,@1648,@1649,@1650,@1651,@1652,@1653,@1654,@1655,@1656,@1657,@1658,@1659,@1660,@1661,@1662,@1663,@1664,@1665,@1666,@1667,@1668,@1669,@1670,@1671,@1672,@1673,@1674,@1675,@1676,@1677,@1678,@1679,@1680,@1681,@1682,@1683,@1684,@1685,@1686,@1687,@1688,@1689,@1690,@1691,@1692,@1693,@1694,@1695,@1696,@1697,@1698,@1699,@1700,@1701,@1702,@1703,@1704,@1705,@1706,@1707,@1708,@1709,@1710,@1711,@1712,@1713,@1714,@1715,@1716,@1717,@1718,@1719,@1720,@1721,@1722,@1723,@1724,@1725,@1726,@1727,@1728,@1729,@1730,@1731,@1732,@1733,@1734,@1735,@1736,@1737,@1738,@1739,@1740,@1741,@1742,@1743,@1744,@1745,@1746,@1747,@1748,@1749,@1750,@1751,@1752,@1753,@1754,@1755,@1756,@1757,@1758,@1759,@1760,@1761,@1762,@1763,@1764,@1765,@1766,@1767,@1768,@1769,@1770,@1771,@1772,@1773,@1774,@1775,@1776,@1777,@1778,@1779,@1780,@1781,@1782,@1783,@1784,@1785,@1786,@1787,@1788,@1789,@1790,@1791,@1792,@1793,@1794,@1795,@1796,@1797,@1798,@1799,@1800,@1801,@1802,@1803,@1804,@1805,@1806,@1807,@1808,@1809,@1810,@1811,@1812,@1813,@1814,@1815,@1816,@1817,@1818,@1819,@1820,@1821,@1822,@1823,@1824,@1825,@1826,@1827,@1828,@1829,@1830,@1831,@1832,@1833,@1834,@1835,@1836,@1837,@1838,@1839,@1840,@1841,@1842,@1843,@1844,@1845,@1846,@1847,@1848,@1849,@1850,@1851,@1852,@1853,@1854,@1855,@1856,@1857,@1858,@1859,@1860,@1861,@1862,@1863,@1864,@1865,@1866,@1867,@1868,@1869,@1870,@1871,@1872,@1873,@1874,@1875,@1876,@1877,@1878,@1879,@1880,@1881,@1882,@1883,@1884,@1885,@1886,@1887,@1888,@1889,@1890,@1891,@1892,@1893,@1894,@1895,@1896,@1897,@1898,@1899,@1900,@1901,@1902,@1903,@1904,@1905,@1906,@1907,@1908,@1909,@1910,@1911,@1912,@1913,@1914,@1915,@1916,@1917,@1918,@1919,@1920,@1921,@1922,@1923,@1924,@1925,@1926,@1927,@1928,@1929,@1930,@1931,@1932,@1933,@1934,@1935,@1936,@1937,@1938,@1939,@1940,@1941,@1942,@1943,@1944,@1945,@1946,@1947,@1948,@1949,@1950,@1951,@1952,@1953,@1954,@1955,@1956,@1957,@1958,@1959,@1960,@1961,@1962,@1963,@1964,@1965,@1966,@1967,@1968,@1969,@1970,@1971,@1972,@1973,@1974,@1975,@1976,@1977,@1978,@1979,@1980,@1981,@1982,@1983,@1984,@1985,@1986,@1987,@1988,@1989,@1990,@1991,@1992,@1993,@1994,@1995,@1996,@1997,@1998,@1999))',N'@0 int,@1 int,@2 int,@3 int,@4 int,@5 int,@6 int,@7 int,@8 int,@9 int,@10 int,@11 int,@12 int,@13 int,@14 int,@15 int,@16 int,@17 int,@18 int,@19 int,@20 int,@21 int,@22 int,@23 int,@24 int,@25 int,@26 int,@27 int,@28 int,@29 int,@30 int,@31 int,@32 int,@33 int,@34 int,@35 int,@36 int,@37 int,@38 int,@39 int,@40 int,@41 int,@42 int,@43 int,@44 int,@45 int,@46 int,@47 int,@48 int,@49 int,@50 int,@51 int,@52 int,@53 int,@54 int,@55 int,@56 int,@57 int,@58 int,@59 int,@60 int,@61 int,@62 int,@63 int,@64 int,@65 int,@66 int,@67 int,@68 int,@69 int,@70 int,@71 int,@72 int,@73 int,@74 int,@75 int,@76 int,@77 int,@78 int,@79 int,@80 int,@81 int,@82 int,@83 int,@84 int,@85 int,@86 int,@87 int,@88 int,@89 int,@90 int,@91 int,@92 int,@93 int,@94 int,@95 int,@96 int,@97 int,@98 int,@99 int,@100 int,@101 int,@102 int,@103 int,@104 int,@105 int,@106 int,@107 int,@108 int,@109 int,@110 int,@111 int,@112 int,@113 int,@114 int,@115 int,@116 int,@117 int,@118 int,@119 int,@120 int,@121 int,@122 int,@123 int,@124 int,@125 int,@126 int,@127 int,@128 int,@129 int,@130 int,@131 int,@132 int,@133 int,@134 int,@135 int,@136 int,@137 int,@138 int,@139 int,@140 int,@141 int,@142 int,@143 int,@144 int,@145 int,@146 int,@147 int,@148 int,@149 int,@150 int,@151 int,@152 int,@153 int,@154 int,@155 int,@156 int,@157 int,@158 int,@159 int,@160 int,@161 int,@162 int,@163 int,@164 int,@165 int,@166 int,@167 int,@168 int,@169 int,@170 int,@171 int,@172 int,@173 int,@174 int,@175 int,@176 int,@177 int,@178 int,@179 int,@180 int,@181 int,@182 int,@183 int,@184 int,@185 int,@186 int,@187 int,@188 int,@189 int,@190 int,@191 int,@192 int,@193 int,@194 int,@195 int,@196 int,@197 int,@198 int,@199 int,@200 int,@201 int,@202 int,@203 int,@204 int,@205 int,@206 int,@207 int,@208 int,@209 int,@210 int,@211 int,@212 int,@213 int,@214 int,@215 int,@216 int,@217 int,@218 int,@219 int,@220 int,@221 int,@222 int,@223 int,@224 int,@225 int,@226 int,@227 int,@228 int,@229 int,@230 int,@231 int,@232 int,@233 int,@234 int,@235 int,@236 int,@237 int,@238 int,@239 int,@240 int,@241 int,@242 int,@243 int,@244 int,@245 int,@246 int,@247 int,@248 int,@249 int,@250 int,@251 int,@252 int,@253 int,@254 int,@255 int,@256 int,@257 int,@258 int,@259 int,@260 int,@261 int,@262 int,@263 int,@264 int,@265 int,@266 int,@267 int,@268 int,@269 int,@270 int,@271 int,@272 int,@273 int,@274 int,@275 int,@276 int,@277 int,@278 int,@279 int,@280 int,@281 int,@282 int,@283 int,@284 int,@285 int,@286 int,@287 int,@288 int,@289 int,@290 int,@291 int,@292 int,@293 int,@294 int,@295 int,@296 int,@297 int,@298 int,@299 int,@300 int,@301 int,@302 int,@303 int,@304 int,@305 int,@306 int,@307 int,@308 int,@309 int,@310 int,@311 int,@312 int,@313 int,@314 int,@315 int,@316 int,@317 int,@318 int,@319 int,@320 int,@321 int,@322 int,@323 int,@324 int,@325 int,@326 int,@327 int,@328 int,@329 int,@330 int,@331 int,@332 int,@333 int,@334 int,@335 int,@336 int,@337 int,@338 int,@339 int,@340 int,@341 int,@342 int,@343 int,@344 int,@345 int,@346 int,@347 int,@348 int,@349 int,@350 int,@351 int,@352 int,@353 int,@354 int,@355 int,@356 int,@357 int,@358 int,@359 int,@360 int,@361 int,@362 int,@363 int,@364 int,@365 int,@366 int,@367 int,@368 int,@369 int,@370 int,@371 int,@372 int,@373 int,@374 int,@375 int,@376 int,@377 int,@378 int,@379 int,@380 int,@381 int,@382 int,@383 int,@384 int,@385 int,@386 int,@387 int,@388 int,@389 int,@390 int,@391 int,@392 int,@393 int,@394 int,@395 int,@396 int,@397 int,@398 int,@399 int,@400 int,@401 int,@402 int,@403 int,@404 int,@405 int,@406 int,@407 int,@408 int,@409 int,@410 int,@411 int,@412 int,@413 int,@414 int,@415 int,@416 int,@417 int,@418 int,@419 int,@420 int,@421 int,@422 int,@423 int,@424 int,@425 int,@426 int,@427 int,@428 int,@429 int,@430 int,@431 int,@432 int,@433 int,@434 int,@435 int,@436 int,@437 int,@438 int,@439 int,@440 int,@441 int,@442 int,@443 int,@444 int,@445 int,@446 int,@447 int,@448 int,@449 int,@450 int,@451 int,@452 int,@453 int,@454 int,@455 int,@456 int,@457 int,@458 int,@459 int,@460 int,@461 int,@462 int,@463 int,@464 int,@465 int,@466 int,@467 int,@468 int,@469 int,@470 int,@471 int,@472 int,@473 int,@474 int,@475 int,@476 int,@477 int,@478 int,@479 int,@480 int,@481 int,@482 int,@483 int,@484 int,@485 int,@486 int,@487 int,@488 int,@489 int,@490 int,@491 int,@492 int,@493 int,@494 int,@495 int,@496 int,@497 int,@498 int,@499 int,@500 int,@501 int,@502 int,@503 int,@504 int,@505 int,@506 int,@507 int,@508 int,@509 int,@510 int,@511 int,@512 int,@513 int,@514 int,@515 int,@516 int,@517 int,@518 int,@519 int,@520 int,@521 int,@522 int,@523 int,@524 int,@525 int,@526 int,@527 int,@528 int,@529 int,@530 int,@531 int,@532 int,@533 int,@534 int,@535 int,@536 int,@537 int,@538 int,@539 int,@540 int,@541 int,@542 int,@543 int,@544 int,@545 int,@546 int,@547 int,@548 int,@549 int,@550 int,@551 int,@552 int,@553 int,@554 int,@555 int,@556 int,@557 int,@558 int,@559 int,@560 int,@561 int,@562 int,@563 int,@564 int,@565 int,@566 int,@567 int,@568 int,@569 int,@570 int,@571 int,@572 int,@573 int,@574 int,@575 int,@576 int,@577 int,@578 int,@579 int,@580 int,@581 int,@582 int,@583 int,@584 int,@585 int,@586 int,@587 int,@588 int,@589 int,@590 int,@591 int,@592 int,@593 int,@594 int,@595 int,@596 int,@597 int,@598 int,@599 int,@600 int,@601 int,@602 int,@603 int,@604 int,@605 int,@606 int,@607 int,@608 int,@609 int,@610 int,@611 int,@612 int,@613 int,@614 int,@615 int,@616 int,@617 int,@618 int,@619 int,@620 int,@621 int,@622 int,@623 int,@624 int,@625 int,@626 int,@627 int,@628 int,@629 int,@630 int,@631 int,@632 int,@633 int,@634 int,@635 int,@636 int,@637 int,@638 int,@639 int,@640 int,@641 int,@642 int,@643 int,@644 int,@645 int,@646 int,@647 int,@648 int,@649 int,@650 int,@651 int,@652 int,@653 int,@654 int,@655 int,@656 int,@657 int,@658 int,@659 int,@660 int,@661 int,@662 int,@663 int,@664 int,@665 int,@666 int,@667 int,@668 int,@669 int,@670 int,@671 int,@672 int,@673 int,@674 int,@675 int,@676 int,@677 int,@678 int,@679 int,@680 int,@681 int,@682 int,@683 int,@684 int,@685 int,@686 int,@687 int,@688 int,@689 int,@690 int,@691 int,@692 int,@693 int,@694 int,@695 int,@696 int,@697 int,@698 int,@699 int,@700 int,@701 int,@702 int,@703 int,@704 int,@705 int,@706 int,@707 int,@708 int,@709 int,@710 int,@711 int,@712 int,@713 int,@714 int,@715 int,@716 int,@717 int,@718 int,@719 int,@720 int,@721 int,@722 int,@723 int,@724 int,@725 int,@726 int,@727 int,@728 int,@729 int,@730 int,@731 int,@732 int,@733 int,@734 int,@735 int,@736 int,@737 int,@738 int,@739 int,@740 int,@741 int,@742 int,@743 int,@744 int,@745 int,@746 int,@747 int,@748 int,@749 int,@750 int,@751 int,@752 int,@753 int,@754 int,@755 int,@756 int,@757 int,@758 int,@759 int,@760 int,@761 int,@762 int,@763 int,@764 int,@765 int,@766 int,@767 int,@768 int,@769 int,@770 int,@771 int,@772 int,@773 int,@774 int,@775 int,@776 int,@777 int,@778 int,@779 int,@780 int,@781 int,@782 int,@783 int,@784 int,@785 int,@786 int,@787 int,@788 int,@789 int,@790 int,@791 int,@792 int,@793 int,@794 int,@795 int,@796 int,@797 int,@798 int,@799 int,@800 int,@801 int,@802 int,@803 int,@804 int,@805 int,@806 int,@807 int,@808 int,@809 int,@810 int,@811 int,@812 int,@813 int,@814 int,@815 int,@816 int,@817 int,@818 int,@819 int,@820 int,@821 int,@822 int,@823 int,@824 int,@825 int,@826 int,@827 int,@828 int,@829 int,@830 int,@831 int,@832 int,@833 int,@834 int,@835 int,@836 int,@837 int,@838 int,@839 int,@840 int,@841 int,@842 int,@843 int,@844 int,@845 int,@846 int,@847 int,@848 int,@849 int,@850 int,@851 int,@852 int,@853 int,@854 int,@855 int,@856 int,@857 int,@858 int,@859 int,@860 int,@861 int,@862 int,@863 int,@864 int,@865 int,@866 int,@867 int,@868 int,@869 int,@870 int,@871 int,@872 int,@873 int,@874 int,@875 int,@876 int,@877 int,@878 int,@879 int,@880 int,@881 int,@882 int,@883 int,@884 int,@885 int,@886 int,@887 int,@888 int,@889 int,@890 int,@891 int,@892 int,@893 int,@894 int,@895 int,@896 int,@897 int,@898 int,@899 int,@900 int,@901 int,@902 int,@903 int,@904 int,@905 int,@906 int,@907 int,@908 int,@909 int,@910 int,@911 int,@912 int,@913 int,@914 int,@915 int,@916 int,@917 int,@918 int,@919 int,@920 int,@921 int,@922 int,@923 int,@924 int,@925 int,@926 int,@927 int,@928 int,@929 int,@930 int,@931 int,@932 int,@933 int,@934 int,@935 int,@936 int,@937 int,@938 int,@939 int,@940 int,@941 int,@942 int,@943 int,@944 int,@945 int,@946 int,@947 int,@948 int,@949 int,@950 int,@951 int,@952 int,@953 int,@954 int,@955 int,@956 int,@957 int,@958 int,@959 int,@960 int,@961 int,@962 int,@963 int,@964 int,@965 int,@966 int,@967 int,@968 int,@969 int,@970 int,@971 int,@972 int,@973 int,@974 int,@975 int,@976 int,@977 int,@978 int,@979 int,@980 int,@981 int,@982 int,@983 int,@984 int,@985 int,@986 int,@987 int,@988 int,@989 int,@990 int,@991 int,@992 int,@993 int,@994 int,@995 int,@996 int,@997 int,@998 int,@999 int,@1000 int,@1001 int,@1002 int,@1003 int,@1004 int,@1005 int,@1006 int,@1007 int,@1008 int,@1009 int,@1010 int,@1011 int,@1012 int,@1013 int,@1014 int,@1015 int,@1016 int,@1017 int,@1018 int,@1019 int,@1020 int,@1021 int,@1022 int,@1023 int,@1024 int,@1025 int,@1026 int,@1027 int,@1028 int,@1029 int,@1030 int,@1031 int,@1032 int,@1033 int,@1034 int,@1035 int,@1036 int,@1037 int,@1038 int,@1039 int,@1040 int,@1041 int,@1042 int,@1043 int,@1044 int,@1045 int,@1046 int,@1047 int,@1048 int,@1049 int,@1050 int,@1051 int,@1052 int,@1053 int,@1054 int,@1055 int,@1056 int,@1057 int,@1058 int,@1059 int,@1060 int,@1061 int,@1062 int,@1063 int,@1064 int,@1065 int,@1066 int,@1067 int,@1068 int,@1069 int,@1070 int,@1071 int,@1072 int,@1073 int,@1074 int,@1075 int,@1076 int,@1077 int,@1078 int,@1079 int,@1080 int,@1081 int,@1082 int,@1083 int,@1084 int,@1085 int,@1086 int,@1087 int,@1088 int,@1089 int,@1090 int,@1091 int,@1092 int,@1093 int,@1094 int,@1095 int,@1096 int,@1097 int,@1098 int,@1099 int,@1100 int,@1101 int,@1102 int,@1103 int,@1104 int,@1105 int,@1106 int,@1107 int,@1108 int,@1109 int,@1110 int,@1111 int,@1112 int,@1113 int,@1114 int,@1115 int,@1116 int,@1117 int,@1118 int,@1119 int,@1120 int,@1121 int,@1122 int,@1123 int,@1124 int,@1125 int,@1126 int,@1127 int,@1128 int,@1129 int,@1130 int,@1131 int,@1132 int,@1133 int,@1134 int,@1135 int,@1136 int,@1137 int,@1138 int,@1139 int,@1140 int,@1141 int,@1142 int,@1143 int,@1144 int,@1145 int,@1146 int,@1147 int,@1148 int,@1149 int,@1150 int,@1151 int,@1152 int,@1153 int,@1154 int,@1155 int,@1156 int,@1157 int,@1158 int,@1159 int,@1160 int,@1161 int,@1162 int,@1163 int,@1164 int,@1165 int,@1166 int,@1167 int,@1168 int,@1169 int,@1170 int,@1171 int,@1172 int,@1173 int,@1174 int,@1175 int,@1176 int,@1177 int,@1178 int,@1179 int,@1180 int,@1181 int,@1182 int,@1183 int,@1184 int,@1185 int,@1186 int,@1187 int,@1188 int,@1189 int,@1190 int,@1191 int,@1192 int,@1193 int,@1194 int,@1195 int,@1196 int,@1197 int,@1198 int,@1199 int,@1200 int,@1201 int,@1202 int,@1203 int,@1204 int,@1205 int,@1206 int,@1207 int,@1208 int,@1209 int,@1210 int,@1211 int,@1212 int,@1213 int,@1214 int,@1215 int,@1216 int,@1217 int,@1218 int,@1219 int,@1220 int,@1221 int,@1222 int,@1223 int,@1224 int,@1225 int,@1226 int,@1227 int,@1228 int,@1229 int,@1230 int,@1231 int,@1232 int,@1233 int,@1234 int,@1235 int,@1236 int,@1237 int,@1238 int,@1239 int,@1240 int,@1241 int,@1242 int,@1243 int,@1244 int,@1245 int,@1246 int,@1247 int,@1248 int,@1249 int,@1250 int,@1251 int,@1252 int,@1253 int,@1254 int,@1255 int,@1256 int,@1257 int,@1258 int,@1259 int,@1260 int,@1261 int,@1262 int,@1263 int,@1264 int,@1265 int,@1266 int,@1267 int,@1268 int,@1269 int,@1270 int,@1271 int,@1272 int,@1273 int,@1274 int,@1275 int,@1276 int,@1277 int,@1278 int,@1279 int,@1280 int,@1281 int,@1282 int,@1283 int,@1284 int,@1285 int,@1286 int,@1287 int,@1288 int,@1289 int,@1290 int,@1291 int,@1292 int,@1293 int,@1294 int,@1295 int,@1296 int,@1297 int,@1298 int,@1299 int,@1300 int,@1301 int,@1302 int,@1303 int,@1304 int,@1305 int,@1306 int,@1307 int,@1308 int,@1309 int,@1310 int,@1311 int,@1312 int,@1313 int,@1314 int,@1315 int,@1316 int,@1317 int,@1318 int,@1319 int,@1320 int,@1321 int,@1322 int,@1323 int,@1324 int,@1325 int,@1326 int,@1327 int,@1328 int,@1329 int,@1330 int,@1331 int,@1332 int,@1333 int,@1334 int,@1335 int,@1336 int,@1337 int,@1338 int,@1339 int,@1340 int,@1341 int,@1342 int,@1343 int,@1344 int,@1345 int,@1346 int,@1347 int,@1348 int,@1349 int,@1350 int,@1351 int,@1352 int,@1353 int,@1354 int,@1355 int,@1356 int,@1357 int,@1358 int,@1359 int,@1360 int,@1361 int,@1362 int,@1363 int,@1364 int,@1365 int,@1366 int,@1367 int,@1368 int,@1369 int,@1370 int,@1371 int,@1372 int,@1373 int,@1374 int,@1375 int,@1376 int,@1377 int,@1378 int,@1379 int,@1380 int,@1381 int,@1382 int,@1383 int,@1384 int,@1385 int,@1386 int,@1387 int,@1388 int,@1389 int,@1390 int,@1391 int,@1392 int,@1393 int,@1394 int,@1395 int,@1396 int,@1397 int,@1398 int,@1399 int,@1400 int,@1401 int,@1402 int,@1403 int,@1404 int,@1405 int,@1406 int,@1407 int,@1408 int,@1409 int,@1410 int,@1411 int,@1412 int,@1413 int,@1414 int,@1415 int,@1416 int,@1417 int,@1418 int,@1419 int,@1420 int,@1421 int,@1422 int,@1423 int,@1424 int,@1425 int,@1426 int,@1427 int,@1428 int,@1429 int,@1430 int,@1431 int,@1432 int,@1433 int,@1434 int,@1435 int,@1436 int,@1437 int,@1438 int,@1439 int,@1440 int,@1441 int,@1442 int,@1443 int,@1444 int,@1445 int,@1446 int,@1447 int,@1448 int,@1449 int,@1450 int,@1451 int,@1452 int,@1453 int,@1454 int,@1455 int,@1456 int,@1457 int,@1458 int,@1459 int,@1460 int,@1461 int,@1462 int,@1463 int,@1464 int,@1465 int,@1466 int,@1467 int,@1468 int,@1469 int,@1470 int,@1471 int,@1472 int,@1473 int,@1474 int,@1475 int,@1476 int,@1477 int,@1478 int,@1479 int,@1480 int,@1481 int,@1482 int,@1483 int,@1484 int,@1485 int,@1486 int,@1487 int,@1488 int,@1489 int,@1490 int,@1491 int,@1492 int,@1493 int,@1494 int,@1495 int,@1496 int,@1497 int,@1498 int,@1499 int,@1500 int,@1501 int,@1502 int,@1503 int,@1504 int,@1505 int,@1506 int,@1507 int,@1508 int,@1509 int,@1510 int,@1511 int,@1512 int,@1513 int,@1514 int,@1515 int,@1516 int,@1517 int,@1518 int,@1519 int,@1520 int,@1521 int,@1522 int,@1523 int,@1524 int,@1525 int,@1526 int,@1527 int,@1528 int,@1529 int,@1530 int,@1531 int,@1532 int,@1533 int,@1534 int,@1535 int,@1536 int,@1537 int,@1538 int,@1539 int,@1540 int,@1541 int,@1542 int,@1543 int,@1544 int,@1545 int,@1546 int,@1547 int,@1548 int,@1549 int,@1550 int,@1551 int,@1552 int,@1553 int,@1554 int,@1555 int,@1556 int,@1557 int,@1558 int,@1559 int,@1560 int,@1561 int,@1562 int,@1563 int,@1564 int,@1565 int,@1566 int,@1567 int,@1568 int,@1569 int,@1570 int,@1571 int,@1572 int,@1573 int,@1574 int,@1575 int,@1576 int,@1577 int,@1578 int,@1579 int,@1580 int,@1581 int,@1582 int,@1583 int,@1584 int,@1585 int,@1586 int,@1587 int,@1588 int,@1589 int,@1590 int,@1591 int,@1592 int,@1593 int,@1594 int,@1595 int,@1596 int,@1597 int,@1598 int,@1599 int,@1600 int,@1601 int,@1602 int,@1603 int,@1604 int,@1605 int,@1606 int,@1607 int,@1608 int,@1609 int,@1610 int,@1611 int,@1612 int,@1613 int,@1614 int,@1615 int,@1616 int,@1617 int,@1618 int,@1619 int,@1620 int,@1621 int,@1622 int,@1623 int,@1624 int,@1625 int,@1626 int,@1627 int,@1628 int,@1629 int,@1630 int,@1631 int,@1632 int,@1633 int,@1634 int,@1635 int,@1636 int,@1637 int,@1638 int,@1639 int,@1640 int,@1641 int,@1642 int,@1643 int,@1644 int,@1645 int,@1646 int,@1647 int,@1648 int,@1649 int,@1650 int,@1651 int,@1652 int,@1653 int,@1654 int,@1655 int,@1656 int,@1657 int,@1658 int,@1659 int,@1660 int,@1661 int,@1662 int,@1663 int,@1664 int,@1665 int,@1666 int,@1667 int,@1668 int,@1669 int,@1670 int,@1671 int,@1672 int,@1673 int,@1674 int,@1675 int,@1676 int,@1677 int,@1678 int,@1679 int,@1680 int,@1681 int,@1682 int,@1683 int,@1684 int,@1685 int,@1686 int,@1687 int,@1688 int,@1689 int,@1690 int,@1691 int,@1692 int,@1693 int,@1694 int,@1695 int,@1696 int,@1697 int,@1698 int,@1699 int,@1700 int,@1701 int,@1702 int,@1703 int,@1704 int,@1705 int,@1706 int,@1707 int,@1708 int,@1709 int,@1710 int,@1711 int,@1712 int,@1713 int,@1714 int,@1715 int,@1716 int,@1717 int,@1718 int,@1719 int,@1720 int,@1721 int,@1722 int,@1723 int,@1724 int,@1725 int,@1726 int,@1727 int,@1728 int,@1729 int,@1730 int,@1731 int,@1732 int,@1733 int,@1734 int,@1735 int,@1736 int,@1737 int,@1738 int,@1739 int,@1740 int,@1741 int,@1742 int,@1743 int,@1744 int,@1745 int,@1746 int,@1747 int,@1748 int,@1749 int,@1750 int,@1751 int,@1752 int,@1753 int,@1754 int,@1755 int,@1756 int,@1757 int,@1758 int,@1759 int,@1760 int,@1761 int,@1762 int,@1763 int,@1764 int,@1765 int,@1766 int,@1767 int,@1768 int,@1769 int,@1770 int,@1771 int,@1772 int,@1773 int,@1774 int,@1775 int,@1776 int,@1777 int,@1778 int,@1779 int,@1780 int,@1781 int,@1782 int,@1783 int,@1784 int,@1785 int,@1786 int,@1787 int,@1788 int,@1789 int,@1790 int,@1791 int,@1792 int,@1793 int,@1794 int,@1795 int,@1796 int,@1797 int,@1798 int,@1799 int,@1800 int,@1801 int,@1802 int,@1803 int,@1804 int,@1805 int,@1806 int,@1807 int,@1808 int,@1809 int,@1810 int,@1811 int,@1812 int,@1813 int,@1814 int,@1815 int,@1816 int,@1817 int,@1818 int,@1819 int,@1820 int,@1821 int,@1822 int,@1823 int,@1824 int,@1825 int,@1826 int,@1827 int,@1828 int,@1829 int,@1830 int,@1831 int,@1832 int,@1833 int,@1834 int,@1835 int,@1836 int,@1837 int,@1838 int,@1839 int,@1840 int,@1841 int,@1842 int,@1843 int,@1844 int,@1845 int,@1846 int,@1847 int,@1848 int,@1849 int,@1850 int,@1851 int,@1852 int,@1853 int,@1854 int,@1855 int,@1856 int,@1857 int,@1858 int,@1859 int,@1860 int,@1861 int,@1862 int,@1863 int,@1864 int,@1865 int,@1866 int,@1867 int,@1868 int,@1869 int,@1870 int,@1871 int,@1872 int,@1873 int,@1874 int,@1875 int,@1876 int,@1877 int,@1878 int,@1879 int,@1880 int,@1881 int,@1882 int,@1883 int,@1884 int,@1885 int,@1886 int,@1887 int,@1888 int,@1889 int,@1890 int,@1891 int,@1892 int,@1893 int,@1894 int,@1895 int,@1896 int,@1897 int,@1898 int,@1899 int,@1900 int,@1901 int,@1902 int,@1903 int,@1904 int,@1905 int,@1906 int,@1907 int,@1908 int,@1909 int,@1910 int,@1911 int,@1912 int,@1913 int,@1914 int,@1915 int,@1916 int,@1917 int,@1918 int,@1919 int,@1920 int,@1921 int,@1922 int,@1923 int,@1924 int,@1925 int,@1926 int,@1927 int,@1928 int,@1929 int,@1930 int,@1931 int,@1932 int,@1933 int,@1934 int,@1935 int,@1936 int,@1937 int,@1938 int,@1939 int,@1940 int,@1941 int,@1942 int,@1943 int,@1944 int,@1945 int,@1946 int,@1947 int,@1948 int,@1949 int,@1950 int,@1951 int,@1952 int,@1953 int,@1954 int,@1955 int,@1956 int,@1957 int,@1958 int,@1959 int,@1960 int,@1961 int,@1962 int,@1963 int,@1964 int,@1965 int,@1966 int,@1967 int,@1968 int,@1969 int,@1970 int,@1971 int,@1972 int,@1973 int,@1974 int,@1975 int,@1976 int,@1977 int,@1978 int,@1979 int,@1980 int,@1981 int,@1982 int,@1983 int,@1984 int,@1985 int,@1986 int,@1987 int,@1988 int,@1989 int,@1990 int,@1991 int,@1992 int,@1993 int,@1994 int,@1995 int,@1996 int,@1997 int,@1998 int,@1999 int',@0=95498,@1=95499,@2=95500,@3=95501,@4=95502,@5=95503,@6=95504,@7=95505,@8=95506,@9=95507,@10=95508,@11=120653,@12=120629,@13=118045,@14=115830,@15=103690,@16=95512,@17=119947,@18=119188,@19=107235,@20=104835,@21=119948,@22=119189,@23=118048,@24=103692,@25=104836,@26=95522,@27=118049,@28=95524,@29=95526,@30=95525,@31=95527,@32=115817,@33=115816,@34=118054,@35=115838,@36=118053,@37=115837,@38=119804,@39=118050,@40=118712,@41=118051,@42=116709,@43=115884,@44=103698,@45=95539,@46=111220,@47=95541,@48=95543,@49=95542,@50=112465,@51=112464,@52=119110,@53=112055,@54=95548,@55=115845,@56=115035,@57=118301,@58=118052,@59=119949,@60=119263,@61=119734,@62=119733,@63=108435,@64=95554,@65=104837,@66=95556,@67=95558,@68=95557,@69=102649,@70=95560,@71=95561,@72=95562,@73=95564,@74=95563,@75=112162,@76=95566,@77=104838,@78=95568,@79=119109,@80=103701,@81=113544,@82=104839,@83=95574,@84=113278,@85=114398,@86=114301,@87=120507,@88=120440,@89=113831,@90=113662,@91=104840,@92=113589,@93=113547,@94=115045,@95=113718,@96=114461,@97=114093,@98=106289,@99=115331,@100=115330,@101=113722,@102=113721,@103=95593,@104=120298,@105=119201,@106=115589,@107=113553,@108=113551,@109=104842,@110=113689,@111=113586,@112=120449,@113=120448,@114=120455,@115=120453,@116=120457,@117=120456,@118=106644,@119=106184,@120=120469,@121=120460,@122=118321,@123=118320,@124=114946,@125=114945,@126=95611,@127=103711,@128=103712,@129=113584,@130=113583,@131=120317,@132=120314,@133=95620,@134=95621,@135=113431,@136=104843,@137=95624,@138=113706,@139=113615,@140=113704,@141=113576,@142=117216,@143=113702,@144=119053,@145=119051,@146=120309,@147=113668,@148=113457,@149=113456,@150=120508,@151=120240,@152=113566,@153=112659,@154=103718,@155=113591,@156=96236,@157=119143,@158=118173,@159=95646,@160=95645,@161=95648,@162=95647,@163=95650,@164=95649,@165=95652,@166=95651,@167=95654,@168=95653,@169=107916,@170=107915,@171=107919,@172=107918,@173=107922,@174=107921,@175=109038,@176=109037,@177=108578,@178=108577,@179=110413,@180=110412,@181=109403,@182=109402,@183=109674,@184=109401,@185=109675,@186=109405,@187=110627,@188=110626,@189=110407,@190=110406,@191=110405,@192=110404,@193=111860,@194=111859,@195=111858,@196=111857,@197=112491,@198=112490,@199=112489,@200=112488,@201=95656,@202=95655,@203=95658,@204=95657,@205=95660,@206=95659,@207=95662,@208=95661,@209=95664,@210=95663,@211=95666,@212=95665,@213=95668,@214=95667,@215=95670,@216=95669,@217=95672,@218=95671,@219=95674,@220=95673,@221=95676,@222=95675,@223=95678,@224=95677,@225=95680,@226=95679,@227=95682,@228=95681,@229=95684,@230=95683,@231=95686,@232=95685,@233=95688,@234=95687,@235=95690,@236=95689,@237=95692,@238=95691,@239=95694,@240=95693,@241=95696,@242=95695,@243=95698,@244=95697,@245=95700,@246=95699,@247=95702,@248=95701,@249=95704,@250=95703,@251=95706,@252=95705,@253=95708,@254=95707,@255=95710,@256=95709,@257=103720,@258=95712,@259=103721,@260=95714,@261=103722,@262=95716,@263=95718,@264=95717,@265=95720,@266=95719,@267=95722,@268=95721,@269=95724,@270=95723,@271=95726,@272=95725,@273=95728,@274=95727,@275=95730,@276=95729,@277=95732,@278=95731,@279=95734,@280=95733,@281=95736,@282=95735,@283=95738,@284=95737,@285=95740,@286=95739,@287=95742,@288=95741,@289=103723,@290=95746,@291=95748,@292=95747,@293=95750,@294=95749,@295=95752,@296=95751,@297=95754,@298=95753,@299=95756,@300=95755,@301=95758,@302=95757,@303=95760,@304=95759,@305=95762,@306=95761,@307=95764,@308=95763,@309=95766,@310=95765,@311=95768,@312=95767,@313=95770,@314=95769,@315=95772,@316=95771,@317=95774,@318=95773,@319=95776,@320=95775,@321=95778,@322=95777,@323=95780,@324=95779,@325=95782,@326=95781,@327=95784,@328=95783,@329=95786,@330=95785,@331=95788,@332=95787,@333=95790,@334=95789,@335=95792,@336=95791,@337=95794,@338=95793,@339=95796,@340=95795,@341=95798,@342=95797,@343=95800,@344=95799,@345=95802,@346=95801,@347=95804,@348=95803,@349=95806,@350=95805,@351=95808,@352=95807,@353=95810,@354=95809,@355=95812,@356=95811,@357=95814,@358=95813,@359=95816,@360=95815,@361=95818,@362=95817,@363=95820,@364=95819,@365=107909,@366=107908,@367=107907,@368=107906,@369=95822,@370=95821,@371=95824,@372=95823,@373=95826,@374=95825,@375=95828,@376=95827,@377=95830,@378=95829,@379=95832,@380=95831,@381=95834,@382=95833,@383=95836,@384=95835,@385=95838,@386=95837,@387=95840,@388=95839,@389=95842,@390=95841,@391=95844,@392=95843,@393=95846,@394=95845,@395=95848,@396=95847,@397=95850,@398=95849,@399=95852,@400=95851,@401=95854,@402=95853,@403=95856,@404=95855,@405=95858,@406=95857,@407=95860,@408=95859,@409=95862,@410=95861,@411=95864,@412=95863,@413=95866,@414=95865,@415=95868,@416=95867,@417=95870,@418=95869,@419=95872,@420=95871,@421=95874,@422=95873,@423=95876,@424=95875,@425=95878,@426=95877,@427=95880,@428=95879,@429=95884,@430=95883,@431=95886,@432=95885,@433=95888,@434=95887,@435=95890,@436=95889,@437=95892,@438=95891,@439=95894,@440=95893,@441=95896,@442=95895,@443=95898,@444=95897,@445=95900,@446=95899,@447=95902,@448=95901,@449=95904,@450=95903,@451=95906,@452=95905,@453=95908,@454=95907,@455=95910,@456=95909,@457=95912,@458=95911,@459=95914,@460=95913,@461=95916,@462=95915,@463=95918,@464=95917,@465=95920,@466=95919,@467=95922,@468=95921,@469=95924,@470=95923,@471=95926,@472=95925,@473=95928,@474=95927,@475=95930,@476=95929,@477=95932,@478=95931,@479=95934,@480=95933,@481=95936,@482=95935,@483=95938,@484=95937,@485=95940,@486=95939,@487=95942,@488=95941,@489=95944,@490=95943,@491=95946,@492=95945,@493=95948,@494=95947,@495=95950,@496=95949,@497=95952,@498=95951,@499=95954,@500=95953,@501=95956,@502=95955,@503=95958,@504=95957,@505=95960,@506=95959,@507=95962,@508=95961,@509=95964,@510=95963,@511=95966,@512=95965,@513=95968,@514=95967,@515=95970,@516=95969,@517=95972,@518=95971,@519=95974,@520=95973,@521=95976,@522=95975,@523=95978,@524=95977,@525=95980,@526=95979,@527=95982,@528=95981,@529=95984,@530=95983,@531=95986,@532=95985,@533=95988,@534=95987,@535=95990,@536=95989,@537=95992,@538=95991,@539=95994,@540=95993,@541=95996,@542=95995,@543=95998,@544=95997,@545=96000,@546=95999,@547=96002,@548=96001,@549=96004,@550=96003,@551=96006,@552=96005,@553=96008,@554=96007,@555=96010,@556=96009,@557=96012,@558=96011,@559=96014,@560=96013,@561=96016,@562=96015,@563=96018,@564=96017,@565=96020,@566=96019,@567=96022,@568=96021,@569=96024,@570=96023,@571=96026,@572=96025,@573=96028,@574=96027,@575=96030,@576=96029,@577=96032,@578=96031,@579=96034,@580=96033,@581=96036,@582=96035,@583=96038,@584=96037,@585=96040,@586=96039,@587=96042,@588=96041,@589=96044,@590=96043,@591=96046,@592=96045,@593=96048,@594=96047,@595=96050,@596=96049,@597=96052,@598=96051,@599=96054,@600=96053,@601=96056,@602=96055,@603=96058,@604=96057,@605=96060,@606=96059,@607=96062,@608=96061,@609=96064,@610=96063,@611=96066,@612=96065,@613=96068,@614=96067,@615=96070,@616=96069,@617=96072,@618=96071,@619=96074,@620=96073,@621=96076,@622=96075,@623=96078,@624=96077,@625=96080,@626=96079,@627=96082,@628=96081,@629=96084,@630=96083,@631=96086,@632=96085,@633=96088,@634=96087,@635=96090,@636=96089,@637=96092,@638=96091,@639=96094,@640=96093,@641=96096,@642=96095,@643=96098,@644=96097,@645=96100,@646=96099,@647=96102,@648=96101,@649=96104,@650=96103,@651=96106,@652=96105,@653=96108,@654=96107,@655=96110,@656=96109,@657=96112,@658=96111,@659=96114,@660=96113,@661=96116,@662=96115,@663=96118,@664=96117,@665=96120,@666=96119,@667=96122,@668=96121,@669=96124,@670=96123,@671=96126,@672=96125,@673=96128,@674=96127,@675=96130,@676=96129,@677=96132,@678=96131,@679=96134,@680=96133,@681=96136,@682=96135,@683=96138,@684=96137,@685=96140,@686=96139,@687=96142,@688=96141,@689=96144,@690=96143,@691=96146,@692=96145,@693=96148,@694=96147,@695=96150,@696=96149,@697=96152,@698=96151,@699=96154,@700=96153,@701=96156,@702=96155,@703=96158,@704=96157,@705=96160,@706=96159,@707=96162,@708=96161,@709=96164,@710=96163,@711=96166,@712=96165,@713=96168,@714=96167,@715=96170,@716=96169,@717=96172,@718=96171,@719=96174,@720=96173,@721=96176,@722=96175,@723=96178,@724=96177,@725=96180,@726=96179,@727=96182,@728=96181,@729=96184,@730=96183,@731=96186,@732=96185,@733=96188,@734=96187,@735=103724,@736=96190,@737=96192,@738=96191,@739=96194,@740=96193,@741=96196,@742=96195,@743=96198,@744=96197,@745=96200,@746=96199,@747=96202,@748=96201,@749=96204,@750=96203,@751=96206,@752=96205,@753=96208,@754=96207,@755=96210,@756=96209,@757=96212,@758=96211,@759=96214,@760=96213,@761=96216,@762=96215,@763=96218,@764=96217,@765=96220,@766=96219,@767=96222,@768=96221,@769=96224,@770=96223,@771=96226,@772=96225,@773=96228,@774=96227,@775=96230,@776=96229,@777=96232,@778=96231,@779=96234,@780=96233,@781=113172,@782=113171,@783=113174,@784=113173,@785=113175,@786=113170,@787=115020,@788=115019,@789=116480,@790=116479,@791=116478,@792=116477,@793=117699,@794=117698,@795=117924,@796=117708,@797=117925,@798=117734,@799=117712,@800=117711,@801=117920,@802=117710,@803=117921,@804=117739,@805=118153,@806=117790,@807=117922,@808=117794,@809=117944,@810=117923,@811=117716,@812=117715,@813=117914,@814=117714,@815=117915,@816=117718,@817=117916,@818=117738,@819=118154,@820=117792,@821=119140,@822=117917,@823=117918,@824=117809,@825=118155,@826=117919,@827=118735,@828=118732,@829=118738,@830=118734,@831=118737,@832=118736,@833=118740,@834=118739,@835=118742,@836=118741,@837=117722,@838=117721,@839=117724,@840=117723,@841=118753,@842=118751,@843=117895,@844=117725,@845=117896,@846=117728,@847=117897,@848=117741,@849=117945,@850=117898,@851=118156,@852=117899,@853=118157,@854=117900,@855=118158,@856=117901,@857=118159,@858=117902,@859=117903,@860=117838,@861=117904,@862=117840,@863=119141,@864=117905,@865=117906,@866=117844,@867=117907,@868=117847,@869=118161,@870=117908,@871=117909,@872=117851,@873=118162,@874=117910,@875=118163,@876=117911,@877=117912,@878=117859,@879=117913,@880=117872,@881=118744,@882=118743,@883=118749,@884=118746,@885=118750,@886=118748,@887=118754,@888=118752,@889=118756,@890=118755,@891=118758,@892=118757,@893=119478,@894=119477,@895=119738,@896=119735,@897=117732,@898=117731,@899=117892,@900=117730,@901=117893,@902=117743,@903=117894,@904=117804,@905=117929,@906=117928,@907=117802,@908=117801,@909=117946,@910=117890,@911=117891,@912=117818,@913=117881,@914=117880,@915=118169,@916=117927,@917=120668,@918=120667,@919=120511,@920=120510,@921=117813,@922=117812,@923=117888,@924=117878,@925=118168,@926=117889,@927=117882,@928=117814,@929=117883,@930=117816,@931=118160,@932=117884,@933=118166,@934=117885,@935=118167,@936=117886,@937=117887,@938=117877,@939=118165,@940=118164,@941=117933,@942=117932,@943=118170,@944=117931,@945=117937,@946=117936,@947=118171,@948=117935,@949=117260,@950=117259,@951=118469,@952=113730,@953=112758,@954=107137,@955=113608,@956=112759,@957=96245,@958=115050,@959=120439,@960=114646,@961=113557,@962=113459,@963=120311,@964=120310,@965=119568,@966=117186,@967=113560,@968=113465,@969=113724,@970=113617,@971=113727,@972=113726,@973=115051,@974=115049,@975=120060,@976=96249,@977=96251,@978=96250,@979=96253,@980=96252,@981=103728,@982=103167,@983=112610,@984=103729,@985=106212,@986=106201,@987=106441,@988=106247,@989=107310,@990=107232,@991=96259,@992=96258,@993=102660,@994=96261,@995=103170,@996=103169,@997=103731,@998=103730,@999=103732,@1000=102663,@1001=107390,@1002=103733,@1003=103734,@1004=102665,@1005=96273,@1006=96272,@1007=103735,@1008=103179,@1009=118082,@1010=103736,@1011=117659,@1012=109662,@1013=103185,@1014=103184,@1015=103738,@1016=102669,@1017=102670,@1018=96285,@1019=103739,@1020=102671,@1021=112608,@1022=103740,@1023=96291,@1024=96290,@1025=103741,@1026=102673,@1027=102674,@1028=96295,@1029=103742,@1030=103191,@1031=103743,@1032=102675,@1033=103744,@1034=102676,@1035=96303,@1036=96302,@1037=102677,@1038=96305,@1039=103194,@1040=103193,@1041=103745,@1042=103197,@1043=103746,@1044=103200,@1045=103747,@1046=102681,@1047=96315,@1048=96314,@1049=115037,@1050=102682,@1051=103748,@1052=103203,@1053=103749,@1054=103206,@1055=102685,@1056=96323,@1057=115036,@1058=102686,@1059=96326,@1060=96328,@1061=96327,@1062=115042,@1063=102687,@1064=115041,@1065=103750,@1066=96334,@1067=96333,@1068=103212,@1069=103211,@1070=115040,@1071=103751,@1072=103752,@1073=102691,@1074=103753,@1075=96342,@1076=103215,@1077=103214,@1078=115039,@1079=103754,@1080=115038,@1081=103755,@1082=112137,@1083=103756,@1084=96352,@1085=96351,@1086=103757,@1087=103221,@1088=102697,@1089=96356,@1090=117660,@1091=109659,@1092=102699,@1093=96360,@1094=102700,@1095=96362,@1096=106970,@1097=106969,@1098=106978,@1099=106968,@1100=107012,@1101=107011,@1102=107296,@1103=107295,@1104=107386,@1105=107385,@1106=107388,@1107=107384,@1108=107598,@1109=107597,@1110=107625,@1111=107604,@1112=107727,@1113=107682,@1114=107815,@1115=107814,@1116=107837,@1117=107830,@1118=107848,@1119=107846,@1120=107894,@1121=107853,@1122=114255,@1123=107949,@1124=107967,@1125=107966,@1126=108125,@1127=108104,@1128=112005,@1129=111464,@1130=108383,@1131=108379,@1132=109655,@1133=108433,@1134=108500,@1135=108499,@1136=108646,@1137=108514,@1138=108588,@1139=108559,@1140=108570,@1141=108568,@1142=112004,@1143=108681,@1144=108845,@1145=108806,@1146=109014,@1147=109013,@1148=96364,@1149=96363,@1150=96366,@1151=96365,@1152=102701,@1153=96368,@1154=102702,@1155=96370,@1156=103759,@1157=96372,@1158=103760,@1159=103227,@1160=96376,@1161=96375,@1162=117648,@1163=109654,@1164=103762,@1165=102704,@1166=107487,@1167=103763,@1168=106290,@1169=103764,@1170=96386,@1171=96385,@1172=103765,@1173=102707,@1174=103766,@1175=103233,@1176=103236,@1177=103235,@1178=103767,@1179=103239,@1180=96396,@1181=96395,@1182=103242,@1183=103241,@1184=96400,@1185=96399,@1186=103768,@1187=103245,@1188=103769,@1189=96404,@1190=102712,@1191=96406,@1192=103770,@1193=102713,@1194=103771,@1195=96410,@1196=103251,@1197=103250,@1198=103772,@1199=102714,@1200=102715,@1201=96416,@1202=103254,@1203=103253,@1204=96420,@1205=96419,@1206=103773,@1207=102717,@1208=117037,@1209=103774,@1210=102719,@1211=96426,@1212=96428,@1213=96427,@1214=103775,@1215=102720,@1216=103257,@1217=103256,@1218=103776,@1219=102721,@1220=96436,@1221=96435,@1222=103777,@1223=102722,@1224=103778,@1225=103260,@1226=103779,@1227=102724,@1228=103780,@1229=102725,@1230=103263,@1231=103262,@1232=96448,@1233=96447,@1234=103266,@1235=103265,@1236=103781,@1237=102727,@1238=103782,@1239=96454,@1240=103783,@1241=102728,@1242=103784,@1243=102729,@1244=96460,@1245=96459,@1246=103785,@1247=102730,@1248=117662,@1249=117661,@1250=103269,@1251=103268,@1252=103787,@1253=103272,@1254=96470,@1255=96469,@1256=103788,@1257=103275,@1258=102735,@1259=96474,@1260=103789,@1261=96476,@1262=103790,@1263=102736,@1264=103791,@1265=102737,@1266=102738,@1267=96482,@1268=102739,@1269=96484,@1270=103792,@1271=102740,@1272=96488,@1273=96487,@1274=103793,@1275=102741,@1276=103794,@1277=102742,@1278=103795,@1279=102743,@1280=102744,@1281=96496,@1282=103796,@1283=102745,@1284=103799,@1285=103798,@1286=103800,@1287=102747,@1288=96504,@1289=96503,@1290=103801,@1291=102748,@1292=103802,@1293=102749,@1294=103803,@1295=103278,@1296=103804,@1297=102750,@1298=112611,@1299=106363,@1300=96516,@1301=96515,@1302=103806,@1303=96518,@1304=102752,@1305=96520,@1306=96522,@1307=96521,@1308=96524,@1309=96523,@1310=103807,@1311=102753,@1312=103808,@1313=102754,@1314=102755,@1315=96530,@1316=102756,@1317=96532,@1318=103809,@1319=96534,@1320=103810,@1321=103281,@1322=103284,@1323=103283,@1324=103287,@1325=103286,@1326=96542,@1327=96541,@1328=103811,@1329=103290,@1330=106348,@1331=103812,@1332=103813,@1333=103293,@1334=103814,@1335=102762,@1336=103815,@1337=102763,@1338=96554,@1339=96553,@1340=103816,@1341=102764,@1342=103817,@1343=102765,@1344=103296,@1345=103295,@1346=103818,@1347=102767,@1348=96564,@1349=96563,@1350=102768,@1351=96566,@1352=112138,@1353=112074,@1354=103299,@1355=103298,@1356=96572,@1357=96571,@1358=103819,@1359=102770,@1360=103820,@1361=102771,@1362=102772,@1363=96578,@1364=103302,@1365=103301,@1366=102774,@1367=96582,@1368=103821,@1369=103305,@1370=103822,@1371=102776,@1372=96588,@1373=96587,@1374=103823,@1375=102777,@1376=102778,@1377=96592,@1378=102779,@1379=96594,@1380=102780,@1381=96596,@1382=102781,@1383=96598,@1384=103824,@1385=102782,@1386=102783,@1387=96602,@1388=102784,@1389=96604,@1390=103825,@1391=102785,@1392=102786,@1393=96608,@1394=96610,@1395=96609,@1396=103826,@1397=102787,@1398=102788,@1399=96614,@1400=102789,@1401=96616,@1402=103827,@1403=102790,@1404=111376,@1405=103828,@1406=102792,@1407=96622,@1408=103829,@1409=102793,@1410=102794,@1411=96626,@1412=102795,@1413=96628,@1414=102796,@1415=96630,@1416=96632,@1417=96631,@1418=102797,@1419=96634,@1420=103830,@1421=102798,@1422=102799,@1423=96638,@1424=103831,@1425=102800,@1426=103832,@1427=102801,@1428=103833,@1429=102802,@1430=102803,@1431=96646,@1432=96648,@1433=96647,@1434=110230,@1435=103834,@1436=103835,@1437=102805,@1438=102806,@1439=96654,@1440=102807,@1441=96656,@1442=102808,@1443=96658,@1444=102809,@1445=96660,@1446=103836,@1447=102810,@1448=96664,@1449=96663,@1450=102811,@1451=96666,@1452=102812,@1453=96668,@1454=103837,@1455=96670,@1456=103838,@1457=96672,@1458=102813,@1459=96674,@1460=103839,@1461=102814,@1462=96678,@1463=96677,@1464=103840,@1465=102815,@1466=102816,@1467=96682,@1468=102817,@1469=96684,@1470=103841,@1471=102818,@1472=102819,@1473=96688,@1474=102820,@1475=96690,@1476=102821,@1477=96692,@1478=102822,@1479=96694,@1480=96696,@1481=96695,@1482=103842,@1483=102823,@1484=102824,@1485=96700,@1486=102825,@1487=96702,@1488=102826,@1489=96704,@1490=103843,@1491=102827,@1492=102828,@1493=96708,@1494=96710,@1495=96709,@1496=96712,@1497=96711,@1498=103844,@1499=102829,@1500=103845,@1501=96716,@1502=102830,@1503=96718,@1504=102831,@1505=96720,@1506=102832,@1507=96722,@1508=102833,@1509=96724,@1510=102834,@1511=96726,@1512=103846,@1513=102835,@1514=102836,@1515=96730,@1516=102837,@1517=96732,@1518=96734,@1519=96733,@1520=102838,@1521=96736,@1522=103847,@1523=102839,@1524=103848,@1525=102840,@1526=102841,@1527=96742,@1528=96744,@1529=96743,@1530=102842,@1531=96746,@1532=102843,@1533=96748,@1534=103849,@1535=96750,@1536=103850,@1537=102844,@1538=96754,@1539=96753,@1540=102845,@1541=96756,@1542=102846,@1543=96758,@1544=103851,@1545=102847,@1546=102848,@1547=96762,@1548=102849,@1549=96764,@1550=102850,@1551=96766,@1552=103852,@1553=102851,@1554=103853,@1555=102852,@1556=96772,@1557=96771,@1558=102853,@1559=96774,@1560=96776,@1561=96775,@1562=102854,@1563=96778,@1564=96780,@1565=96779,@1566=102855,@1567=96782,@1568=103854,@1569=102856,@1570=102857,@1571=96786,@1572=102858,@1573=96788,@1574=103855,@1575=102859,@1576=102860,@1577=96792,@1578=96794,@1579=96793,@1580=102861,@1581=96796,@1582=102862,@1583=96798,@1584=96800,@1585=96799,@1586=102863,@1587=96802,@1588=102864,@1589=96804,@1590=106334,@1591=103856,@1592=103857,@1593=102866,@1594=102867,@1595=96810,@1596=103858,@1597=102868,@1598=103859,@1599=102869,@1600=103860,@1601=102870,@1602=96818,@1603=96817,@1604=102871,@1605=96820,@1606=103861,@1607=102872,@1608=102873,@1609=96824,@1610=102874,@1611=96826,@1612=103862,@1613=102875,@1614=102876,@1615=96830,@1616=102877,@1617=96832,@1618=96834,@1619=96833,@1620=103863,@1621=102878,@1622=102879,@1623=96838,@1624=102880,@1625=96840,@1626=108552,@1627=103864,@1628=103865,@1629=102882,@1630=102883,@1631=96846,@1632=102884,@1633=96848,@1634=103866,@1635=102885,@1636=102886,@1637=96852,@1638=102887,@1639=96854,@1640=96856,@1641=96855,@1642=96858,@1643=96857,@1644=103867,@1645=102888,@1646=102889,@1647=96862,@1648=103868,@1649=102890,@1650=102891,@1651=96866,@1652=102892,@1653=96868,@1654=102893,@1655=96870,@1656=102894,@1657=96872,@1658=96874,@1659=96873,@1660=103869,@1661=102895,@1662=102896,@1663=96878,@1664=102897,@1665=96880,@1666=102898,@1667=96882,@1668=102899,@1669=96884,@1670=103870,@1671=102900,@1672=96888,@1673=96887,@1674=96890,@1675=96889,@1676=103871,@1677=102901,@1678=103872,@1679=102902,@1680=96896,@1681=96895,@1682=96898,@1683=96897,@1684=102903,@1685=96900,@1686=102904,@1687=96902,@1688=116355,@1689=102905,@1690=103873,@1691=102906,@1692=102907,@1693=96908,@1694=102908,@1695=96910,@1696=103874,@1697=102909,@1698=96914,@1699=96913,@1700=102910,@1701=96916,@1702=102911,@1703=96918,@1704=102912,@1705=96920,@1706=103875,@1707=102913,@1708=102914,@1709=96924,@1710=103876,@1711=102915,@1712=102916,@1713=96928,@1714=102917,@1715=96930,@1716=103877,@1717=102918,@1718=96934,@1719=96933,@1720=102919,@1721=96936,@1722=102920,@1723=96938,@1724=102921,@1725=96940,@1726=103878,@1727=102922,@1728=96944,@1729=96943,@1730=102923,@1731=96946,@1732=102924,@1733=96948,@1734=102925,@1735=96950,@1736=102926,@1737=96952,@1738=102927,@1739=96954,@1740=96956,@1741=96955,@1742=103879,@1743=102928,@1744=103880,@1745=102929,@1746=103881,@1747=102930,@1748=103882,@1749=102931,@1750=102932,@1751=96966,@1752=103883,@1753=102933,@1754=102934,@1755=96970,@1756=102935,@1757=96972,@1758=96974,@1759=96973,@1760=103884,@1761=102936,@1762=103885,@1763=102937,@1764=103886,@1765=102938,@1766=103887,@1767=102939,@1768=103888,@1769=102940,@1770=103889,@1771=102941,@1772=103890,@1773=102942,@1774=103891,@1775=102943,@1776=102944,@1777=96992,@1778=103892,@1779=102945,@1780=96996,@1781=96995,@1782=102946,@1783=96998,@1784=102947,@1785=97000,@1786=102948,@1787=97002,@1788=102949,@1789=97004,@1790=102950,@1791=97006,@1792=103893,@1793=102951,@1794=103894,@1795=102952,@1796=102953,@1797=97012,@1798=97014,@1799=97013,@1800=102954,@1801=97016,@1802=103895,@1803=97018,@1804=102955,@1805=97020,@1806=102956,@1807=97022,@1808=97024,@1809=97023,@1810=103896,@1811=97026,@1812=102957,@1813=97028,@1814=102958,@1815=97030,@1816=102959,@1817=97032,@1818=103897,@1819=102960,@1820=102961,@1821=97036,@1822=97038,@1823=97037,@1824=102962,@1825=97040,@1826=103898,@1827=102963,@1828=107233,@1829=107110,@1830=102965,@1831=97046,@1832=102966,@1833=97048,@1834=102967,@1835=97050,@1836=102968,@1837=97052,@1838=103899,@1839=102969,@1840=103900,@1841=102970,@1842=103901,@1843=102971,@1844=103902,@1845=102972,@1846=97062,@1847=97061,@1848=102973,@1849=97064,@1850=102974,@1851=97066,@1852=102975,@1853=97068,@1854=103903,@1855=102976,@1856=102977,@1857=97072,@1858=97074,@1859=97073,@1860=103904,@1861=102978,@1862=103905,@1863=102979,@1864=102980,@1865=97080,@1866=103906,@1867=102981,@1868=102982,@1869=97084,@1870=97086,@1871=97085,@1872=97088,@1873=97087,@1874=102983,@1875=97090,@1876=103907,@1877=102984,@1878=102985,@1879=97094,@1880=102986,@1881=97096,@1882=102987,@1883=97098,@1884=97100,@1885=97099,@1886=102988,@1887=97102,@1888=97104,@1889=97103,@1890=97106,@1891=97105,@1892=97108,@1893=97107,@1894=103908,@1895=102989,@1896=102990,@1897=97112,@1898=102991,@1899=97114,@1900=102992,@1901=97116,@1902=102993,@1903=97118,@1904=97120,@1905=97119,@1906=103909,@1907=102994,@1908=102995,@1909=97124,@1910=102996,@1911=97126,@1912=102997,@1913=97128,@1914=102998,@1915=97130,@1916=102999,@1917=97132,@1918=97134,@1919=97133,@1920=103000,@1921=97136,@1922=103910,@1923=97138,@1924=103911,@1925=103001,@1926=103912,@1927=103002,@1928=103003,@1929=97144,@1930=103913,@1931=103004,@1932=103914,@1933=103005,@1934=103006,@1935=97150,@1936=103915,@1937=103007,@1938=103916,@1939=103008,@1940=97156,@1941=97155,@1942=103009,@1943=97158,@1944=103917,@1945=103010,@1946=103918,@1947=103011,@1948=103919,@1949=103012,@1950=103013,@1951=97166,@1952=97168,@1953=97167,@1954=103920,@1955=97170,@1956=103921,@1957=97172,@1958=107106,@1959=103922,@1960=97176,@1961=97175,@1962=103923,@1963=97178,@1964=103924,@1965=97180,@1966=103925,@1967=97182,@1968=103926,@1969=97184,@1970=103927,@1971=97186,@1972=103928,@1973=97188,@1974=107111,@1975=103929,@1976=103930,@1977=97192,@1978=97194,@1979=97193,@1980=103931,@1981=97196,@1982=103932,@1983=97198,@1984=103933,@1985=97200,@1986=103934,@1987=97202,@1988=103935,@1989=97204,@1990=97206,@1991=97205,@1992=103936,@1993=97208,@1994=103937,@1995=97210,@1996=103938,@1997=97212,@1998=103939,@1999=97214

[russellshome]

"This is a bugger of a query"

Umbraco-CMS/src/Umbraco.Infrastructure/Persistence/Repositories/Implement/ContentRepositoryBase.cs

Line 755 in d9dced5

// TODO: This is a bugger of a query and I believe is the main issue with regards to SQL performance drain when querying content

Perhaps this issue can be solved by this query not necessitating the locking of the umbracoLock table while doing that query?

[Zeegaan]

@russellshome If you can reproduce it every time, could I get the database from you? Would help greatly in debugging the issue 👍

[russellshome]

Do you have a location where I can put it?

[Zeegaan]

You can send it to my email: nge@umbraco.dk 👌
A backpack file of the database would be great 🙌

[russellshome]

Are you sure your email will accept close to a a GB file? Most email servers will reject that sort of size. I don't think I will be able to send it let alone you receive it attached to an email.

[chriskarkowsky]

Hi,

I have the same issue running 11.4.0.

I have increased the WriteLockTimeout to 20 seconds from 5:
"DistributedLockingWriteLockDefaultTimeout": "00:00:20"

After trying to Save, I always get the error:

We are setting up a load balanced environment in accordance with the documentation here: https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/azure-web-apps

We have two applications, one for backoffice and one for the frontend. We have also assigned the proper Roles for each, where frontend would be the subscriber and backoffice would be the scheduler (https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/flexible-advanced).

[chriskarkowsky]

It seems that this query is closely associated with this issue and at times takes an inordinately long time to complete

exec sp_executesql N'UPDATE umbracoLock WITH (REPEATABLEREAD) SET value = (CASE WHEN (value=1) THEN -1 ELSE 1 END) WHERE id=@0',N'@0 int',@0=-333

I noticed this too in Long Running Queries in the Azure Dashboard. It seems like a simple toggle of the value for the ContentTree node.

Whenever the WriteLock issue appears, I tried simply running the toggle via an UPDATE command
UPDATE [dbo].[umbracoLock] set [value]=-1 where [id]=-333

Should be an instantaneous result, but you can see it is taking way over the WriteLock timeout limit:

We have a 50 DTU allocated database, and we are well below max DTU usage.

[andrewmoscardino-awh]

I should chime in here. A project I work on was having this issue in production and the fix seemed to be making sure that the Lucene/Examine settings were correct for the admin and front-end app services.

The docs for running on Azure Web Apps say to set Umbraco.CMS.Examine.LuceneDirectoryFactory to SyncedTempFileSystemDirectoryFactory. What I missed on the docs for load balancing on Azure Web Apps was that only the admin site should use that value. Front-end sites should use TempFileSystemDirectoryFactory.

After that was corrected, the errors stopped.

[chriskarkowsky]

I did a bit more digging into the database, and found that my issue isn't explicitly caused by the umbracoLock table, but the way Azure may be recycling my app with CI/CD. I noticed this after recycling the app from the portal or pushing changes.

After a restart of the Umbraco app (Dockerized), I did the following.

1. Ran "exec sp_who2" in the database to get a list of clients. Noticed there was one "sleeping" for the hostname assigned by Azure for the Core Microsoft Client
2. Once the app booted, I tried saving. First time worked, second time failed for with the write lock error.
3. Running "exec sp_who2" again, I now noticed there were 2 different clients with the same hostname.
4. During clicking save again, I ran sp_who2 and noticed that the one client was SUSPENDED and being blocked by the duplicate client.

^During trying to Save in Umbraco backoffice, before a Write Lock error is thrown.

5. After this, I had to run KILL 84, and then no more write lock error!

I am not sure how to address this issue, whether Umbraco is not correctly closing the connection or with Azure not stopping the Docker image correctly. But thought I would share.

[thomashdk]

Having same issue on 11.4.2.

Have multiple solutions running on same code. But only this solution has this error.

It used around 20-40% more memory then the other solutions. This one don't have any trafic, the others have.
It has a lot of nodes and media files. (94774)

I'm not running on a azure setup up. Just a windows server and a sql server

Running the Data Integrity health check fails with:
Microsoft.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.) ---> System.ComponentModel.Win32Exception (10054): An existing connection was forcibly closed by the remote host.

I have these console errors also:

Is there a fix or a workaround ?

[chriskarkowsky]

Thanks for bumping this at it is still casing issues up to today. The only way I have found to workaround it is to kill the process. This isn't going to work when multiple users are trying to save in the backoffice.

I am still researching what is going on at a DB level during the save.

1. Click Save on a page in the backoffice
2. I run the following SQL cmd immediately
   SELECT * FROM sys.dm_exec_requests CROSS APPLY sys.dm_exec_sql_text(sql_handle)
   

The newly spun up process (69) is being blocked by 54

3. run dbcc inputbuffer(54)
   Event info shows this query:
   (@0 int,@1 int,@2 int,@3 int,@4 int,@5 int,@6 int,@7 int,@8 int)SELECT [cmsPropertyType].[id] AS [Id], [cmsPropertyType].[dataTypeId] AS [DataTypeId], [cmsPropertyType].[contentTypeId] AS [ContentTypeId], [cmsPropertyType].[propertyTypeGroupId] AS [PropertyTypeGroupId], [cmsPropertyType].[Alias] AS [Alias], [cmsPropertyType].[Name] AS [Name], [cmsPropertyType].[sortOrder] AS [SortOrder], [cmsPropertyType].[mandatory] AS [Mandatory], [cmsPropertyType].[mandatoryMessage] AS [MandatoryMessage], [cmsPropertyType].[validationRegExp] AS [ValidationRegExp], [cmsPropertyType].[validationRegExpMessage] AS [ValidationRegExpMessage], [cmsPropertyType].[Description] AS [Description], [cmsPropertyType].[labelOnTop] AS [LabelOnTop], [cmsPropertyType].[variations] AS [Variations], [cmsPropertyType].[UniqueID] AS [UniqueId], [umbracoDataType].[nodeId] AS [DataTypeDto__NodeId], [umbracoDataType].[propertyEditorAlias] AS [DataTypeDto__EditorAlias], [umbracoDataType].[dbType] AS [DataTypeDto__DbType], [umbracoDataType].[config] AS [DataTypeDto__Configuration]FROM [cmsPropertyType]INNER JOIN [umbracoDataType]ON ([cmsPropertyType].[dataTypeId] = [umbracoDataType].[nodeId])WHERE ([cmsPropertyType].[id] IN (@0,@1,@2,@3,@4,@5,@6,@7,@8))

This is similar to @russellshome was seeing.

4. At this point I would manually run KILL 54 to resolve the issue and allow users to continue to save.
5. I should mention that 54 is a sleeping process that needs to be killed. I don't understand how a sleeping process can be blocking the new UPDATE.

[Zeegaan]

@russellshome Sorry for leaving you unanswered for a while, I did not remember this after vacation. 😨

Hmmm if email won't do, I guess a dropbox link or something similar? 🤔

[thomashdk]

Thanks for bumping this at it is still casing issues up to today. The only way I have found to workaround it is to kill the process. This isn't going to work when multiple users are trying to save in the backoffice.

I am still researching what is going on at a DB level during the save.

1. Click Save on a page in the backoffice
2. I run the following SQL cmd immediately
   SELECT * FROM sys.dm_exec_requests CROSS APPLY sys.dm_exec_sql_text(sql_handle)
   

The newly spun up process (69) is being blocked by 54

3. run dbcc inputbuffer(54)
   Event info shows this query:
   (@0 int,@1 int,@2 int,@3 int,@4 int,@5 int,@6 int,@7 int,@8 int)SELECT [cmsPropertyType].[id] AS [Id], [cmsPropertyType].[dataTypeId] AS [DataTypeId], [cmsPropertyType].[contentTypeId] AS [ContentTypeId], [cmsPropertyType].[propertyTypeGroupId] AS [PropertyTypeGroupId], [cmsPropertyType].[Alias] AS [Alias], [cmsPropertyType].[Name] AS [Name], [cmsPropertyType].[sortOrder] AS [SortOrder], [cmsPropertyType].[mandatory] AS [Mandatory], [cmsPropertyType].[mandatoryMessage] AS [MandatoryMessage], [cmsPropertyType].[validationRegExp] AS [ValidationRegExp], [cmsPropertyType].[validationRegExpMessage] AS [ValidationRegExpMessage], [cmsPropertyType].[Description] AS [Description], [cmsPropertyType].[labelOnTop] AS [LabelOnTop], [cmsPropertyType].[variations] AS [Variations], [cmsPropertyType].[UniqueID] AS [UniqueId], [umbracoDataType].[nodeId] AS [DataTypeDto__NodeId], [umbracoDataType].[propertyEditorAlias] AS [DataTypeDto__EditorAlias], [umbracoDataType].[dbType] AS [DataTypeDto__DbType], [umbracoDataType].[config] AS [DataTypeDto__Configuration]FROM [cmsPropertyType]INNER JOIN [umbracoDataType]ON ([cmsPropertyType].[dataTypeId] = [umbracoDataType].[nodeId])WHERE ([cmsPropertyType].[id] IN (@0,@1,@2,@3,@4,@5,@6,@7,@8))

This is similar to @russellshome was seeing.

4. At this point I would manually run KILL 54 to resolve the issue and allow users to continue to save.
5. I should mention that 54 is a sleeping process that needs to be killed. I don't understand how a sleeping process can be blocking the new UPDATE.

Yeah, I'm doing the same for now...

[chriskarkowsky]

Here is a lineage from the log of 4 items when I attempted to refresh the Internal Index and getting write lock, all on the same process (69):

[chriskarkowsky]

Here are a couple more related errors:

[madsoulswe]

This seems to be some Umbraco and AzureSql specific issue. Would really appreciate update from Umbraco team on where this issue is in priority as mentioned above.

We had the same issue on Umbraco 12.x on Ubuntu with Sqlserver, when we tried importing a site using uSync to empty site (150 content, 300 media).

I tried debugging the issue, but it's hard to replicate and understanding what Umbraco is doing is above my level 😅

[remkovb]

Still state/needs-more-info so guess that’s the reason nothing happens with this one.

guess we can conclude with newer comments it’s not especially an azure thing, but there is something fundamentally wrong in some of the sql query’s/functionality that’s used to save a node in SQL. I’m also not able to pinpoint this in umbraco source myself, so hopefully any smart umbraco core developer can investigate this soon.

[remkovb]

@nul800sebastiaan please take a look at this one.. is getting more urgent every week, I guess for a lot of us...

[remkovb]

@andr317c maybe? Saw you closing the related issue.. Do you know when this will be picked up? It's from May 2023, so would be really cool if this will be fixed soon because we don't know how to fix it ourselves ;-)

[thomashdk]

Still a issue in Umbraco 13.1.1

[remkovb]

I dived a bit into this one and found the line where this exception is happening:

https://github.com/umbraco/Umbraco-CMS/blob/contrib/src/Umbraco.Core/Services/ContentVersionService.cs line 179

When publishing node and in the background older versions are being deleted it wants to achieve a write lock, but here it fails.

So in https://github.com/umbraco/Umbraco-CMS/blob/contrib/src/Umbraco.Cms.Persistence.SqlServer/Services/SqlServerDistributedLockingMechanism.cs the DistributedWriteLockTimeoutException happens when creating an instance of SqlServerDistributedLock:

Hopefuly this will help any of the core developers to pinpoint this issue and find out WHY this write lock can't be acquired.

Full exception:
Umbraco.Cms.Core.DistributedLocking.Exceptions.DistributedWriteLockTimeoutException: Failed to acquire write lock for id: -333. at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock..ctor(SqlServerDistributedLockingMechanism parent, Int32 lockId, DistributedLockType lockType, TimeSpan timeout) at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.WriteLock(Int32 lockId, Nullable1 obtainLockTimeout) at Umbraco.Cms.Infrastructure.Scoping.Scope.ObtainWriteLock(Int32 lockId, Nullable1 timeout) at Umbraco.Cms.Infrastructure.Scoping.Scope.LockInner(Guid instanceId, Dictionary2& locks, HashSet1& locksSet, Action2 obtainLock, Nullable1 timeout, Int32 lockId) at Umbraco.Cms.Infrastructure.Scoping.Scope.EagerWriteLockInner(Guid instanceId, Nullable1 timeout, Int32[] lockIds) at Umbraco.Cms.Infrastructure.Scoping.Scope.EnsureDbLocks() at Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database() at Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.DocumentVersionRepository.DeleteVersions(IEnumerable1 versionIds) at Umbraco.Cms.Core.Services.ContentVersionService.CleanupDocumentVersions(DateTime asAtDate) at Umbraco.Cms.Core.Services.ContentVersionService.PerformContentVersionCleanup(DateTime asAtDate) at Umbraco.Cms.Infrastructure.HostedServices.ContentVersionCleanup.PerformExecuteAsync(Object state) at Umbraco.Cms.Infrastructure.HostedServices.RecurringHostedServiceBase.ExecuteAsync(Object state)

[jerpenol]

We are also still experiencing the same issues on the latest version of 13.

[chriskarkowsky]

I have noticed since we have moved to production is that this always occurs when we swap code (using staging slots in Azure). I know for sure that eventually the users in the backoffice will get the error.

We also have two instances, one for the frontend and one backend, and it doesn't matter which one is swapped, the backend will get the error.

We have also attempted waiting after the swap occurs and the error pops to see if it sorts itself out, and we have waited a matter of hours without it resolving, so we go in and kill the process.

I will say though once the instance has been up and running and we are not constantly swapping code, no new instances are establishing db connections, and the app remains stable without restarting itself in Azure, this error does not happen. (Knock on wood) it's been weeks since our last code swap and users have not hit the writelock error in the backoffice.

I am not sure if anyone else is in this position to notice, since we are all probably debugging and restarting constantly. This is still hamstringing development severely, and we have no confidence in scaling the webapp out. We actually had to scale our frontend down because the more instances running, the more processes to kill.

Radio silence on this issue is concerning as I can only imagine it is affecting all .NET Core versions of Umbraco. This issue did not exist in Umbraco 8 (which we migrated from).

[deanparkr]

The radio silence is the most frustrating thing. This has been open for so long and I see in the thread people have tried to reach out to Umbraco staff to comment.

Its having such a large impact on our business and it is really undermining the business confidence that people have in Umbraco as this issue directly impacts end users.

It is a shame there is not a better more defined process for reporting issues to Umbraco. There is no indication right now that this issue is on anyone's radar or how we can move this out of "Needs more info" label.

[remkovb]

Totally agree..

This is a really urgent issues for us and our clients who want answers we can't give..

Maybe we should create a new issue, because it seems this issue is ignored by umbraco staff right now.

[nul800sebastiaan]

We're reading along at Umbraco HQ, there is not a lot of actionable information being added here.

So, status:

• October 2023 Fixes have been made for 10.8.0 and 12.3.0 , but as noted there: "it might need real-world testing before we can confirm it closes Failed to acquire write lock for id: -333 #14195"
• December 8, 2023 - that didn't seem to cover all problems. As noted: "We are still searching for a potential fix, but we have a hard time reproducing."
• In that same comment above, code has been added as a workaround - the person who did test it seems to have had some luck since they didn't report much in the way of problems back
• December 13, 2023 - "We have been searching for a hanging lock for some time" Background job causes "Failed to acquire write lock for id: -333." #15445 (comment) / that thread also notes potential problems with load balanced setups but doesn't come to a conclusion either as it is still difficult to pinpoint

What's next:

At this poing I would encourage everyone to check out the potential workaround as noted in #13804 (comment)
Please report back on the above.

That said, if I read between the lines and write something from gut feelings:
While not specifically anything to do with Azure, it seems to have a lot to do with load balanced scenarios, and please don't immediately say you're not load balancing. As soon as 2 webservers are trying to run Umbraco at the same time, however short this may be, you're effectively load balancing.
So what does that mean?

• Specicifally on Azure, when slot swapping, you might have overlapping app pools for a single Umbraco site, meaning 2 sites are trying to access the same database.
• Specifally in local development scenarios, you mihgt have 2 developers using the same database for an Umbraco site they're working on.
• Specifically for any other hosting scenarios, you might be in a situation where a server is still winding down while the other server is booting up, meaning 2 sites are trying to access the same database.

Of course that doesn't mean that there is no bug, there really seems to be.
So after trying out the workaround for a good amount of time, we will need more information. It's still happening, tell us anything and everything you can think of that might be related, including version numbers please. Read through the related issues thoroughly, see if you can find commonalities with other people.

If anyone can spin up a test site, which demonstrates the problem each and every time then do share that with us, then we can investigate further. As noted a few times, we're having a hard time reliably reproducing the problem.

What will help us move it forward is insightful comments saying what your setup is, which workarounds you've tried and any other clues you might be able to give us.

[nul800sebastiaan]

One additional hint is that it might help to increase DistributedLockingWriteLockDefaultTimeout as well, as noted in #13804 (comment)

[dawoe]

Just stumbled on this thread.

So for the people who are using deployment slots in azure I would strongly setting "MainDomKeyDiscriminator" configuration setting to a unique value per slot. Has to be done in the azure portal.
We were having similar problems, and making this unique solved a lot of problems. We have also configured all our deployment slots as a subscriber instance. So we don't have 2 publishing servers while swapping.

See https://docs.umbraco.com/umbraco-cms/reference/configuration/globalsettings#main-dom-key-discriminator

Dave

[bergmania]

To add a few more info to #14195 (comment)

It's important to emphasise the fact that seeing the message "Failed to acquire write lock for id: -333" is not an error in it self.
It's by design, but maybe a bad error message.
This always happens if another scope reads for a longer period than the DistributedLockingWriteLockDefaultTimeout.

We know that a lot of things are going on when booting, also in background threads. In particular, the potential rebuilding of the examine indexes can be long running processes that hold the lock for some time.

Our locking system allows multiple readers at the same time, but the write lock is exclusive and must wait for all readers to complete, and all subsequent locks wait until it completes. What happens when this error shows up, is that some scope is holding a read or write lock.

What potentially is a bug is the hanging threads, which some have reported.
Furthermore, we are searching for whatever it is that locks the db so long, but haven't found it.

The db reading when rebuilding examine index seems to be already be optimised as much as it can, currently in regards to how early it releases the lock.

Our investigation is bit limited, as we dont know what is having the lock, when a new scope can't get it - And thereby show this meesage. We can't easily add that information to the current locking implementation, as that would effectively turn all read locks into write locks - And thereby make the issue bigger.

[chriskarkowsky]

Thanks @bergmania for the explanation and the update of what your team has investigated.

This always happens if another scope reads for a longer period than the DistributedLockingWriteLockDefaultTimeout.

Our locking system allows multiple readers at the same time, but the write lock is exclusive and must wait for all readers to complete, and all subsequent locks wait until it completes. What happens when this error shows up, is that some scope is holding a read or write lock.

This is interesting as I was not aware that the Write can only occur if there are no ReadLocks. This means that there is not necessarily something else holding the WriteLock, but it could mean that a thread with a ReadLock is stalled and never relinquishing.

As you said, on boot, there is a lot of "stuff" going on, principle of which is the reindexing from Examine. Is this the main focus of queries that hold ReadLocks? From the previous errors I have posted, especially about the Child Scope not being disposed of and the Sql ZombieCheck, the error/problem is likely somewhere within Examine. I have a lot of content/media, so the issue likely scales with the size of the site. Would this be true?

I spun up a new instance of Umbraco with the Starter Kit and am trying to get to a point to reproduce the error. I can only dedicate so much time to this, however. I will update if I find anything, especially a way to consistently reproduce.

[bergmania]

As you said, on boot, there is a lot of "stuff" going on, principle of which is the reindexing from Examine. Is this the main focus of queries that hold ReadLocks? From the previous errors I have posted, especially about the Child Scope not being disposed of and the Sql ZombieCheck, the error/problem is likely somewhere within Examine. I have a lot of content/media, so the issue likely scales with the size of the site. Would this be true?

The issue definitely scale with the size og the site, as all queries becomes a bit slower, the more content you have.

[bergmania]

Umbraco 10.8.5, 12.3.9 & 13.2.2 all contain a potential fix for this. As I mentioned earlier, just seeing this error message is not necessarily a bug and can be by design, but in that case a lock should have waited the for a timeout.

[chriskarkowsky]

@bergmania thank you! I am going to install and start testing 12.3.9.

[thomashdk]

Still getting Lock request time out period exceeded on a solution with many node (119952). Running 13.2.2.

It looks when running Usync import

[bergmania]

@thomashdk, do you have a stacktrace for that one?

[thomashdk]

@bergmania Yeah of course, sorry for not adding that!

Microsoft.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows)
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)
   at Microsoft.Data.SqlClient.SqlCommand.CompleteExecuteScalar(SqlDataReader ds, Boolean returnSqlValue)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteScalar()
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteScalarHelper(DbCommand cmd)
   at NPoco.Database.ExecuteScalar[T](String sql, CommandType commandType, Object[] args)
   at Umbraco.Cms.Infrastructure.Persistence.UmbracoDatabase.ExecuteScalar[T](String sql, Object[] args)
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock.ObtainReadLock()
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock..ctor(SqlServerDistributedLockingMechanism parent, Int32 lockId, DistributedLockType lockType, TimeSpan timeout)
ClientConnectionId:c98c2ed7-59d4-44d2-8ead-ea0be9173173
Error Number:1222,State:51,Class:16

[bergmania]

@thomashdk , I can't read anything interesting from that. Do we know what calls the SqlServerDistributedLock..ctor

[simonech]

Hi @bergmania @nul800sebastiaan
we have migrated our website on Azure and we have huge performance issues:
the frontend is working ok, but the backend is crazy slow.
We have P3V3 as plan and D5 (400DTU) as DB, and all are floating around 15-20% max of CPU or DTUs, but save/publish operations take up to one minute sometimes, and seeing the waterfall chart on execution on AppInsight it looks like there are sometimes 1000s of queries per page and many times there are 10-15 sec between SQL operations, and many times indeed we also get this "failed to get lock" error.
So, indeed, it seems something specific to lock issues, that sometime takes too long...
Extending the timeout might prevent these errors, but the fact that it wait maybe 40sec for acquiring a lock is troublesome.
We contacted support (we have enterprise support), but they told us to upgrade to 13.2.2 and reassess later. But I don't see any improvement here.

I'd really appreciate if you could follow up on this: we have a special partnership between you and my company, so you would have full access to be able to troubleshoot the issue if you wanted, and skipping the tedious "did you restart your computer" kind of questions/suggestions from 1st level support.

[chriskarkowsky]

@bergmania Yeah of course, sorry for not adding that!

Microsoft.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows)
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)
   at Microsoft.Data.SqlClient.SqlCommand.CompleteExecuteScalar(SqlDataReader ds, Boolean returnSqlValue)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteScalar()
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteScalarHelper(DbCommand cmd)
   at NPoco.Database.ExecuteScalar[T](String sql, CommandType commandType, Object[] args)
   at Umbraco.Cms.Infrastructure.Persistence.UmbracoDatabase.ExecuteScalar[T](String sql, Object[] args)
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock.ObtainReadLock()
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock..ctor(SqlServerDistributedLockingMechanism parent, Int32 lockId, DistributedLockType lockType, TimeSpan timeout)
ClientConnectionId:c98c2ed7-59d4-44d2-8ead-ea0be9173173
Error Number:1222,State:51,Class:16

I deployed 12.3.9 to a dev instance webapp and got this error at first, but within a few minutes it stopped happening. I don't know if it was a symptoms of starting up, but I did not need to do my usual KILL process in the database to fix it. I was then able to save pages and rebuild indexes without errors. So so far, this is looking good, but I will continue to test.

Also pertaining to the Event Log, it quickly writes way too many Debug records for locking so that I am not able to view it in the backoffice. I also can not see it on the server as it is locked. Is there a way to disable this? If I did get an error, I wouldn't be able to see it without stopping the app.

Unable to view logs
Today's log file is too large to be viewed and would cause performance problems.

If you need to view the log files, try opening them manually

[KartikGolya24]

Hi @bergmania @nul800sebastiaan
We have migrated our site to the latest version of Umbraco (v13.2.2) but still, we are also facing this same issue Lock request time out period exceeded
Stacktrace:

Microsoft.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
The statement has been terminated.
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock.ObtainWriteLock()
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock..ctor(SqlServerDistributedLockingMechanism parent, Int32 lockId, DistributedLockType lockType, TimeSpan timeout)
ClientConnectionId:da53a57a-a255-47bf-9f70-acadf835da34
Error Number:1222,State:51,Class:16
ClientConnectionId before routing:b0b2e847-bc00-40f4-b208-9848bb565aca
Routing Destination:cf5b56abe9af.tr53379.westeurope1-a.worker.database.windows.net,11056

Screenshots:

[bergmania]

Hi @KartikGolya24

Thanks for input. I can't see anything from the stacktraces, but can you confirm if it happens both at Content save and content save and publish?

[KartikGolya24]

Hi @KartikGolya24

Thanks for input. I can't see anything from the stacktraces, but can you confirm if it happens both at Content save and content save and publish?

@bergmania Yes, it is happening for both. And also while saving any DocumentType or DataType.