TypeError: Cannot read property 'substring' of undefined - RESTART! #2836

micah · 2015-11-17T15:38:49Z

We have a very active set of pads, with several hundred users on them. Unfortunately, something is causing etherpad to restart on a regular basis, this is the log of when it does:

[2015-11-17 07:27:54.247] [WARN] client - Uncaught TypeError: Cannot read property 'data' of undefined -- { errorId: 'ZdIdZ8nXVLQTE9tVbPgb',
  msg: 'Uncaught TypeError: Cannot read property \'data\' of undefined',
  url: '[REDACTED]',
  linenumber: 3902,
  userAgent: 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36' }
[2015-11-17 07:28:13.168] [WARN] client - TypeError: undefined is not an object (evaluating 'rowsPresent[index].data') -- { errorId: '7FGxdP5milNbRykoStKV',
  msg: 'TypeError: undefined is not an object (evaluating \'rowsPresent[index].data\')',
  url: '[REDACTED]',
  linenumber: 3902,
  userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.7 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.7' }
[2015-11-17 07:28:45.256] [WARN] client - SyntaxError: expected expression, got ')' -- { errorId: 'X45Sz4e8TZIuMmRRkoCA',
  msg: 'SyntaxError: expected expression, got \')\'',
  url: '[REDACTED]',
  linenumber: 1,
  userAgent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0' }
[2015-11-17 07:28:46.691] [WARN] client - Uncaught Error: Failed assertion: mismatched apply: 6831 / 6841 -- { errorId: 'I07sb37FnB4hqN8MEU8S',
  msg: 'Uncaught Error: Failed assertion: mismatched apply: 6831 / 6841',
  url: '[REDACTED]',
  linenumber: 492,
  userAgent: 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36' }
[2015-11-17 07:28:46.779] [WARN] client - Error: Failed assertion: mismatched apply: 6831 / 6840 -- { errorId: 'mpoHFiJQyUqHJG6BmXc1',
  msg: 'Error: Failed assertion: mismatched apply: 6831 / 6840',
  url: '[REDACTED]',
  linenumber: 5871,
  userAgent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0' }
[2015-11-17 07:28:56.631] [ERROR] console - TypeError: Cannot read property 'substring' of undefined
    at Object.insert (/srv/etherpad-lite/src/static/js/Changeset.js:794:44)
    at Object.exports.mutateTextLines (/srv/etherpad-lite/src/static/js/Changeset.js:966:11)
    at /srv/etherpad-lite/src/node/handler/PadMessageHandler.js:1511:19
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:610:21
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:249:17
    at iterate (/srv/etherpad-lite/src/node_modules/async/lib/async.js:149:13)
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:160:25
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:251:21
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:615:34
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:254:17
RESTART!

The text was updated successfully, but these errors were encountered:

webzwo0i · 2015-11-17T16:33:49Z

which commit are you using? can you cherry pick #2801

micah · 2015-11-17T17:06:54Z

8359756

micah · 2015-11-17T17:10:51Z

I tried to cherry-pick 1ee1f81 to start, but got a conflict that I was not sure how to resolve. I suspect your version is based off of a different place than the 1.5.7 release?

webzwo0i · 2015-11-17T17:16:27Z

try cherry-picking 31f7c7e before 1ee1f81

micah · 2015-11-17T17:42:56Z

thanks, cherry-picked both of those and restarted the server, will report back

micah · 2015-11-17T17:52:41Z

Just got another restart:

[2015-11-17 09:51:37.247] [ERROR] console - TypeError: Cannot read property 'substring' of undefined
    at Object.insert (/srv/etherpad-lite/src/static/js/Changeset.js:794:44)
    at Object.exports.mutateTextLines (/srv/etherpad-lite/src/static/js/Changeset.js:966:11)
    at /srv/etherpad-lite/src/node/handler/PadMessageHandler.js:1517:19
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:610:21
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:249:17
    at iterate (/srv/etherpad-lite/src/node_modules/async/lib/async.js:149:13)
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:160:25
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:251:21
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:615:34
    at /srv/etherpad-lite/src/node_modules/async/lib/async.js:254:17
RESTART!

webzwo0i · 2015-11-17T18:57:54Z

ok I try to find the bug.
as a workaround you can filter GET requests to /p//timeslider (which will block accessing timeslider and clients from accidently triggering the bug) or insert "return" right at the beginning of handleChangesetRequest in PadMessageHandler.js

micah · 2015-11-17T19:37:39Z

Do you mean in the function, like this:

function handleChangesetRequest(client, message)
{
+  return;
  //check if all ok
  if(message.data == null)
  {
    messageLogger.warn("Dropped message, changeset request has no data!");
    return;
  }

Because I tried that and the timeslider is still working.

webzwo0i · 2015-11-17T19:53:25Z

yep, it's loading timeslider's html but you won't be able to issue requests to the timeslider (which contain the actual revisions) - at least it should work like this imo :)

JohnMcLear · 2015-12-07T13:44:51Z

*bump

webzwo0i · 2015-12-15T14:05:14Z

I can reproduce the bug now but the test data is huge and the area it occurs is not documented. investigating now, but not sure I'll find it today...

joergludwig · 2016-01-11T12:38:09Z

I have got the same problem over here:
[2016-01-11 13:17:46.347] [ERROR] console - TypeError: Cannot call method 'substring' of undefined
at Object.insert (/usr/share/etherpad-lite/src/static/js/Changeset.js:794:45)
at Object.exports.mutateTextLines (/usr/share/etherpad-lite/src/static/js/Changeset.js:966:11)
at async.series.callback.forwardsChangesets (/usr/share/etherpad-lite/src/node/handler/PadMessageHandler.js:1540:19)
at /usr/share/etherpad-lite/src/node_modules/async/lib/async.js:610:21
at /usr/share/etherpad-lite/src/node_modules/async/lib/async.js:249:17
at iterate (/usr/share/etherpad-lite/src/node_modules/async/lib/async.js:149:13)
at /usr/share/etherpad-lite/src/node_modules/async/lib/async.js:160:25
at /usr/share/etherpad-lite/src/node_modules/async/lib/async.js:251:21
at /usr/share/etherpad-lite/src/node_modules/async/lib/async.js:615:34
at /usr/share/etherpad-lite/src/node_modules/async/lib/async.js:254:17

I was able to reproduce the problem on a different server by exporting and importing the affected etherpad. Unfortunately I cannot publish the file as it contains data of our customers.

joergludwig · 2016-01-11T16:01:47Z

I was able to remove all custom text from the etherpad while preserving the error.

Steps to reproduce:

Download and unzip attached file.
Create new pad.
Import etherpad file.
Open history.
Jump back some time.
Etherpad crashes.

They test if it is possible to delete lines from the array that is mutated by a changeset, and after no line is left insert some characters (with or without new lines).

micah · 2016-02-06T15:59:24Z

Just wanted to add some additional missing information here... we were also getting restarts/crashes when the export was done. Not always, but probably with certain pads with characters/character sets that were not filtered properly. This was causing us such a problem that we had to disable pad exports... but this is something people need so we would like to be able to re-enable it!

webzwo0i · 2016-02-15T05:30:36Z

sorry for the delay... for the issue described here I opened a pull request at #2911

RE your export issues please pull from https://github.com/ether/etherpad-lite/tree/fix-export-regressions. It is not ready to be merged in but it should be enough to stop the crashes. It would be awesome if you could try it and if nothing happens grep for "Could not process export request" and send me any warnings you got.

micah · 2016-02-15T16:16:07Z

@webzwo0i - great! I'd love to give them a try, but I'm currently running the latest release (1.5.7), and not the tip of master. Unfortunately, I dont want to move the code to master on the live instance. I tried to cherry-pick the commits onto that release, but it wouldn't apply. Are those commits easy to rebase onto that release?

webzwo0i · 2016-02-17T05:57:11Z

I prepared a patch file for the export fixes at https://gist.github.com/webzwo0i/99c57a5482a0f28eb3e6
You should be able to apply it on 1.5.7 with

git clone https://gist.github.com/webzwo0i/99c57a5482a0f28eb3e6
patch -p1 < 99c57a5482a0f28eb3e6/export-fixes.patch 
rm -rf 99c57a5482a0f28eb3e6

the other fix is not ready to apply, I discovered more issues, but I am working on this and will report back soon

micah · 2016-02-17T19:17:26Z

Thanks @webzwo0i ! I've applied the export fixes patch as described, and am running it now, so far so good... but I'll need to wait a little bit to see if the original issue comes back.

micah · 2016-03-23T13:53:26Z

Since applying that patch from @webzwo0i we haven't experienced that issue!

micah · 2016-03-23T19:31:56Z

Well... I spoke too soon. I had a webserver config line that just denied exports.

chao-mu · 2018-03-13T20:07:43Z

Was a fix for this ever applied? Found the following in my logs and I'm assuming it's a manifestation of this bug:

[2018-03-13 19:26:00.885] [ERROR] console - TypeError: Cannot read property 'substring' of undefined
[2018-03-13 19:26:00.886] [INFO] console - graceful shutdown...

I am running version 1.5.7. Curious if a newer version will fix this.

JohnMcLear · 2018-04-03T17:15:46Z

This seems to be fixed now.

They test if it is possible to delete lines from the array that is mutated by a changeset, and after no line is left insert some characters (with or without new lines).

webzwo0i mentioned this issue Feb 15, 2016

Fix text lines mutator #2911

Closed

JohnMcLear closed this as completed Apr 3, 2018

marcovtwout mentioned this issue Jun 27, 2018

Crash when using time slider #3371

Closed

rhansen pushed a commit that referenced this issue Sep 26, 2021

tests: Add regression tests for #2836

3eea5f3

They test if it is possible to delete lines from the array that is mutated by a changeset, and after no line is left insert some characters (with or without new lines).

rhansen pushed a commit that referenced this issue Sep 26, 2021

tests: Add regression tests for #2836

c84001e

They test if it is possible to delete lines from the array that is mutated by a changeset, and after no line is left insert some characters (with or without new lines).

rhansen pushed a commit that referenced this issue Sep 30, 2021

tests: Add regression tests for #2836

89be875

They test if it is possible to delete lines from the array that is mutated by a changeset, and after no line is left insert some characters (with or without new lines).

rhansen pushed a commit that referenced this issue Oct 17, 2021

tests: Add regression tests for #2836

927a3e7

They test if it is possible to delete lines from the array that is mutated by a changeset, and after no line is left insert some characters (with or without new lines).

webzwo0i added a commit that referenced this issue Oct 30, 2021

easysync: add regression tests for #2836, #5214 and #3560

43de73d

rhansen pushed a commit that referenced this issue Nov 9, 2021

easysync: add regression tests for #2836, #5214 and #3560

9ce7a26

rhansen pushed a commit that referenced this issue Nov 13, 2021

easysync: add regression tests for #2836, #5214 and #3560

399d13d

rhansen pushed a commit that referenced this issue Nov 14, 2021

easysync: add regression tests for #2836, #5214 and #3560

f38f2f3

rhansen pushed a commit that referenced this issue Nov 23, 2021

easysync: add regression tests for #2836, #5214 and #3560

c86b857

rhansen pushed a commit that referenced this issue Nov 23, 2021

easysync: add regression tests for #2836, #5214 and #3560

31c779d

rhansen pushed a commit that referenced this issue Nov 23, 2021

easysync: add regression tests for #2836, #5214 and #3560

9e658a7

rhansen pushed a commit that referenced this issue Nov 24, 2021

easysync: add regression tests for #2836, #5214 and #3560

54b23dd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TypeError: Cannot read property 'substring' of undefined - RESTART! #2836

TypeError: Cannot read property 'substring' of undefined - RESTART! #2836

micah commented Nov 17, 2015

webzwo0i commented Nov 17, 2015

micah commented Nov 17, 2015

micah commented Nov 17, 2015

webzwo0i commented Nov 17, 2015

micah commented Nov 17, 2015

micah commented Nov 17, 2015

webzwo0i commented Nov 17, 2015

micah commented Nov 17, 2015

webzwo0i commented Nov 17, 2015

JohnMcLear commented Dec 7, 2015

webzwo0i commented Dec 15, 2015

joergludwig commented Jan 11, 2016

joergludwig commented Jan 11, 2016

micah commented Feb 6, 2016

webzwo0i commented Feb 15, 2016

micah commented Feb 15, 2016

webzwo0i commented Feb 17, 2016

micah commented Feb 17, 2016

micah commented Mar 23, 2016

micah commented Mar 23, 2016

chao-mu commented Mar 13, 2018 •

edited

JohnMcLear commented Apr 3, 2018

TypeError: Cannot read property 'substring' of undefined - RESTART! #2836

TypeError: Cannot read property 'substring' of undefined - RESTART! #2836

Comments

micah commented Nov 17, 2015

webzwo0i commented Nov 17, 2015

micah commented Nov 17, 2015

micah commented Nov 17, 2015

webzwo0i commented Nov 17, 2015

micah commented Nov 17, 2015

micah commented Nov 17, 2015

webzwo0i commented Nov 17, 2015

micah commented Nov 17, 2015

webzwo0i commented Nov 17, 2015

JohnMcLear commented Dec 7, 2015

webzwo0i commented Dec 15, 2015

joergludwig commented Jan 11, 2016

joergludwig commented Jan 11, 2016

micah commented Feb 6, 2016

webzwo0i commented Feb 15, 2016

micah commented Feb 15, 2016

webzwo0i commented Feb 17, 2016

micah commented Feb 17, 2016

micah commented Mar 23, 2016

micah commented Mar 23, 2016

chao-mu commented Mar 13, 2018 • edited

JohnMcLear commented Apr 3, 2018

chao-mu commented Mar 13, 2018 •

edited