Multiple different driver bugs from a large-scale parse

[vmarkovtsev]

I noticed that the cpp driver crashed at some point of my massive GitHub repository parsing.
Here are the logs: https://drive.google.com/file/d/1SqQt4s6sAJtF5im4nb3nY-Ig1K7gquQ9

[bzz]

It's going to take a while to grok 500mb of logs :) Thank you very much!

From a very quick glance there are at least some:

• 2 known js driver bugs

  • JSXIdentifier: comments are not handled javascript-driver#69 fixed
  • transform failed: rune out of bounds: 2200447 [0, 2200314) javascript-driver#71

• 1 unknown js driver bug

  request processed content 7151 bytes error: rpc error: code = FailedPrecondition desc = received 2 errors:
       check: key "value": invalid syntax
       check: key "value": invalid syntax ('atom-text-editor[data-grammar*=\"js\"].editor .class-6')
  

  semantic: string literal escape sequence handing javascript-driver#75

• 3 unknown PHP driver bugs

  • unused field(s) on node Expr_Include: comments
  • check: unused field(s) on node Scalar_String: [attributes comments]
  • FailedPrecondition desc = offset out of bounds: 5493 [0, 5490]"
    https://github.com/bblfsh/php-driver/milestone/1

• 2 ruby driver bugs

  • rpc error: code = FailedPrecondition desc = column out of bounds: 0 [1, 1]" elapsed=1.30897693s filename="app/models/tender.rb" language=ruby
    

  •  received message larger than max (4485021 vs. 4194304)" elapsed=19.413394312s filename="data/azure_service_fabric/lib/6.2.0.9/generated/azure_service_fabric/service_fabric_client_apis.rb" language=ruby`
    

  Column out of bounds ruby-driver#43
  Grpc: received message larger than max (5797446 vs. 4194304) sdk#384

• 1 golang driver bug received message larger than max (4272334 vs. 4194304)
  Grpc: received message larger than max (5797446 vs. 4194304) sdk#384

• 2 java driver bugs

  • check: key "escapedValue": invalid syntax ("\0")" elapsed=6.154940858s filename="src/main/java/com/android/tools/r8/ir/code/BasicBlock.java
  • received message larger than max (8386845 vs. 4194304)
    https://github.com/bblfsh/java-driver/milestone/1
    Grpc: received message larger than max (5797446 vs. 4194304) sdk#384

• a cpp driver bugs?

  time="2019-03-20T08:40:11Z" level=error msg="request processed content 50971 bytes error: rpc error: code = Internal desc = unexpected end of JSON input" elapsed=1.635467213s filename="tests/unity_internals.h"
  time="2019-03-20T08:41:40Z" level=error msg="request processed content 56 bytes error: rpc error: code = Internal desc = native driver is not running" elapsed=3.291429ms fil
  

  and 25 semantic normalization failures :)
  semantic: multiple transformation failures cpp-driver#31

• a python driver bugs

  • unknown driver crashes
    error: rpc error: code = Internal desc = native driver is not running
  • known driver failure: invalid character 'I' looking for beginning of value python-driver#182
  • known transform failed: column out of bounds: 29 [1, 25] python-driver#181
  • new unknown check: key "returns": unused field(s) on node BoxedName: noops_sameline python-driver#187

Going to dig deeper tomorrow and update the list here, before proceeding with fixes.

[vmarkovtsev]

I am currently running massive parsing on several machines, once they finish I will share more logs 👍

[bzz]

13 bugs so far 😆 - I think it's a fair start :)
Going to dig deeper in each of those, case by case, before doing any more breadth error coverage reporting.

Good news is that the next bblfhsd release going to include a fix for restarting the drivers bblfsh/sdk#381 so it would simplify both, large scale parsing and log error hunting.

[bzz]

Could not resist: here is a nice distribution of error logs per language

cpp        21202
javascript 4076
php        2958
python     1166
csharp     219
java       107
go         86
ruby       20
bash       0
typescript 0

Script to reproduce

for LANG in java javascript php ruby go python csharp cpp bash typescript
do
  echo "${LANG}"; grep "language=${LANG}$" bblfshd.log | grep -c "error"
done

grep -oP "error: rpc error: code = [[:alnum:]]*" bblfshd.log | sort | uniq -c | sort -dr

And here is the distribution of all 34,892 grpc error reports by type

  22292 error: rpc error: code = Internal
  11762 error: rpc error: code = FailedPrecondition
    416 error: rpc error: code = DeadlineExceeded
    394 error: rpc error: code = Canceled
     28 error: rpc error: code = ResourceExhausted

Scripts in details could be useful to compare this with the next log analysis, using a new bblfshd version.

[dennwc]

Should we move this issue to bblfshd then? Sounds like it becomes more an umbrella issue than something specific for C++ driver.

[bzz]

Moved to bblfshd. Going to file individual driver issues from #268 (comment) now.

[creachadair]

here is a nice distribution of error logs per language

A concrete demonstration of Bjarne Stroustrup's occasional observation that there are two kinds of languages: Those everyone complains about, and those nobody uses. Apparently even the compiler agrees. 😜

[bzz]

Triaged, actionable bugs:

• PHP https://github.com/bblfsh/php-driver/milestone/1
• JavaScript semantic: string literal escape sequence handing javascript-driver#75
• SDK Grpc: received message larger than max (5797446 vs. 4194304) sdk#384
• Ruby Column out of bounds ruby-driver#43
• Java https://github.com/bblfsh/java-driver/milestone/1
• Cpp https://github.com/bblfsh/cpp-driver/milestone/1
• Python https://github.com/bblfsh/python-driver/milestone/1

The only 1 thing that is left before we can close this one are: dig a bit deeper to the origin of Python driver restarts e.g by listing the last errors before the first restart.

Update:

• python driver seems to have crushed on driver failure: invalid character 'I' looking for beginning of value python-driver#182 and never recovered
• cpp seems to have crushed with Internal desc = unexpected end of JSON input that most probably is a symptom that @dennwc could help to shed some light on.
  @vmarkovtsev if you could help to identify the project where the tests/unity_internals.h is coming from - that would help a lot.

[bzz]

As soon as origin of last C++ failure is identified and a new issue is created for it under https://github.com/bblfsh/cpp-driver/milestone/1 - I think it should be good to close this issue in favor of individual driver milestones.

[vmarkovtsev]

@bzz FWIW here is another log which is 8 times bigger, taken yesterday night: https://drive.google.com/file/d/1kBCdZm49sVEjVzVmcFX1flUgv6gvHBw1

[bzz]

@bzz FWIW here is another log which is 8 times bigger, taken yesterday night: https://drive.google.com/file/d/1kBCdZm49sVEjVzVmcFX1flUgv6gvHBw1

Thank you! We still have to finish the scope from the previous one first, I'm afraid 😆

What would really help is if we could collect a new log like that as soon as new bblfhsd is released. E.g if you could describe some simple procedure of getting one - we could run it on our own.

Otherwise, I'll get back and analyse this log but after finishing with current issues, most probably next week. Thanks again!

[dennwc]

If it's possible, let's run the same large-scale parse again with the latest bblfshd. It should fix most of the issues.

[vmarkovtsev]

I am going to do this next week.

[bzz]

It seems like all, but C++ and Ruby driver bugs were already fixed now.

[dennwc]

Let's wait for the new bblfshd release, rerun the parse job and see if it catches anything new.

[dennwc]

v2.13.0 is out, we can start testing it.

[bzz]

@dennwc do you think we could close this one now, in favor of a separate new issue for different logs?

[dennwc]

@bzz Yes, sounds reasonable. Created a new one: #298