Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

URLDownloader doesn't work on docker-compose #146

Closed
kam193 opened this issue Nov 24, 2023 · 4 comments
Closed

URLDownloader doesn't work on docker-compose #146

kam193 opened this issue Nov 24, 2023 · 4 comments
Labels
accepted This issue was accepted, we will work on this at some point bug Something isn't working service-urldownloader Related to URLDownloader service

Comments

@kam193
Copy link

kam193 commented Nov 24, 2023 •
edited

Describe the bug
I cannot make URLDownloader service working, and it's hard to get any related logs - neither in the production nor development mode. For any URL I send, the results are:

  • empty (but only when using an older version of the service),
  • The number of retries has passed the limit. & task pre-empted
  • Command Command '['java', ...]' timed out after 150 seconds
  • Exception: Kangooroo was probably OOMKilled. Check for memory usage and increase limit as needed.

I tried to debug the issue by copying the generated config and running Kangooroo manually, to see what's happening. Results are:

Logs from Kangaroo 
java -Dlogback.configurationFile=logback.xml -jar KangoorooStandalone.jar --conf-file /var/tmp/debuging/tmp4iel01_a/tmp5ai10wyz --dont-use-captcha --url https://google.com
15:08:50.331 [main] INFO  c.g.c.k.KangoorooStandaloneRunner - We are NOT using captcha solver.
15:08:50.355 [main] INFO  c.g.c.k.b.KangoorooChromeBrowser - Fetching using Chrome: https://google.com [99999ebcfdb78df077ad2727fd00969f]
15:08:50.716 [main] INFO  o.l.p.impl.DefaultHttpProxyServer - Starting proxy at address: 0.0.0.0/0.0.0.0:0
15:08:50.736 [main] INFO  o.l.p.impl.DefaultHttpProxyServer - Proxy listening with TCP transport
15:08:50.835 [main] INFO  o.l.p.impl.DefaultHttpProxyServer - Proxy started at address: /0.0.0.0:45061
ChromeDriver was started successfully.
15:08:54.414 [main] WARN  c.g.c.k.b.KangoorooChromeBrowser - Unable to start Chrome, trying again..
org.openqa.selenium.SessionNotCreatedException: session not created: Chrome failed to start: exited normally.
  (session not created: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /opt/google/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'f11c03ceabb2', ip: '172.20.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '6.2.0-37-generic', java.version: '11.0.21'
Driver info: driver.version: CustomChromeDriver
remote stacktrace: #0 0x558a5ce465e3 <unknown>
#1 0x558a5cb090b7 <unknown>
#2 0x558a5cb3fe55 <unknown>
#3 0x558a5cb3cb81 <unknown>
#4 0x558a5cb8747f <unknown>
#5 0x558a5cb7dcc3 <unknown>
#6 0x558a5cb490e4 <unknown>
#7 0x558a5cb4a0ae <unknown>
#8 0x558a5ce0cce1 <unknown>
#9 0x558a5ce10b7e <unknown>
#10 0x558a5cdfa4b5 <unknown>
#11 0x558a5ce117d6 <unknown>
#12 0x558a5cddddbf <unknown>
#13 0x558a5ce34748 <unknown>
#14 0x558a5ce34917 <unknown>
#15 0x558a5ce45773 <unknown>
#16 0x7f917e7cafa3 start_thread

        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
        at org.openqa.selenium.remote.W3CHandshakeResponse.lambda$errorHandler$0(W3CHandshakeResponse.java:62) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.HandshakeResponse.lambda$getResponseFunction$0(HandshakeResponse.java:30) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.ProtocolHandshake.lambda$createSession$0(ProtocolHandshake.java:126) ~[KangoorooStandalone.jar:na]
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[na:na]
        at java.base/java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958) ~[na:na]
        at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[na:na]
        at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:na]
        at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:543) ~[na:na]
        at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:128) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:74) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:136) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:213) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:131) ~[KangoorooStandalone.jar:na]
        at ca.gc.cyber.kangooroo.browser.chrome.CustomChromeDriver.<init>(CustomChromeDriver.java:40) ~[KangoorooStandalone.jar:na]
        at ca.gc.cyber.kangooroo.browser.KangoorooChromeBrowser.createDriver(KangoorooChromeBrowser.java:527) ~[KangoorooStandalone.jar:na]
        at ca.gc.cyber.kangooroo.browser.KangoorooChromeBrowser.execute(KangoorooChromeBrowser.java:102) ~[KangoorooStandalone.jar:na]
        at ca.gc.cyber.kangooroo.browser.KangoorooBrowser.get(KangoorooBrowser.java:79) ~[KangoorooStandalone.jar:na]
        at ca.gc.cyber.kangooroo.KangoorooStandaloneRunner.main(KangoorooStandaloneRunner.java:281) ~[KangoorooStandalone.jar:na]
15:09:00.052 [main] WARN  c.g.c.k.b.KangoorooChromeBrowser - Unable to start Chrome, trying again..
org.openqa.selenium.SessionNotCreatedException: session not created: Chrome failed to start: exited normally.
  (session not created: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /opt/google/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'f11c03ceabb2', ip: '172.20.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '6.2.0-37-generic', java.version: '11.0.21'
Driver info: driver.version: CustomChromeDriver
remote stacktrace: #0 0x558a5ce465e3 <unknown>
#1 0x558a5cb090b7 <unknown>
#2 0x558a5cb3fe55 <unknown>
#3 0x558a5cb3cb81 <unknown>
#4 0x558a5cb8747f <unknown>
#5 0x558a5cb7dcc3 <unknown>
#6 0x558a5cb490e4 <unknown>
#7 0x558a5cb4a0ae <unknown>
#8 0x558a5ce0cce1 <unknown>
#9 0x558a5ce10b7e <unknown>
#10 0x558a5cdfa4b5 <unknown>
#11 0x558a5ce117d6 <unknown>
#12 0x558a5cddddbf <unknown>
#13 0x558a5ce34748 <unknown>
#14 0x558a5ce34917 <unknown>
#15 0x558a5ce45773 <unknown>
#16 0x7f917e7cafa3 start_thread

        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
        at org.openqa.selenium.remote.W3CHandshakeResponse.lambda$errorHandler$0(W3CHandshakeResponse.java:62) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.HandshakeResponse.lambda$getResponseFunction$0(HandshakeResponse.java:30) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.ProtocolHandshake.lambda$createSession$0(ProtocolHandshake.java:126) ~[KangoorooStandalone.jar:na]
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[na:na]
        at java.base/java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958) ~[na:na]
        at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[na:na]
        at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:na]
        at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:543) ~[na:na]
        at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:128) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:74) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:136) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:213) ~[KangoorooStandalone.jar:na]
        at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:131) ~[KangoorooStandalone.jar:na]
        at ca.gc.cyber.kangooroo.browser.chrome.CustomChromeDriver.<init>(CustomChromeDriver.java:40) ~[KangoorooStandalone.jar:na]
        at ca.gc.cyber.kangooroo.browser.KangoorooChromeBrowser.createDriver(KangoorooChromeBrowser.java:527) ~[KangoorooStandalone.jar:na]
        at ca.gc.cyber.kangooroo.browser.KangoorooChromeBrowser.execute(KangoorooChromeBrowser.java:102) ~[KangoorooStandalone.jar:na]
        at ca.gc.cyber.kangooroo.browser.KangoorooBrowser.get(KangoorooBrowser.java:79) ~[KangoorooStandalone.jar:na]
        at ca.gc.cyber.kangooroo.KangoorooStandaloneRunner.main(KangoorooStandaloneRunner.java:281) ~[KangoorooStandalone.jar:na]
Exception in thread "main" org.openqa.selenium.SessionNotCreatedException: session not created: Chrome failed to start: exited normally.
  (session not created: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /opt/google/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'f11c03ceabb2', ip: '172.20.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '6.2.0-37-generic', java.version: '11.0.21'
Driver info: driver.version: CustomChromeDriver
remote stacktrace: #0 0x558a5ce465e3 <unknown>
#1 0x558a5cb090b7 <unknown>
#2 0x558a5cb3fe55 <unknown>
#3 0x558a5cb3cb81 <unknown>
#4 0x558a5cb8747f <unknown>
#5 0x558a5cb7dcc3 <unknown>
#6 0x558a5cb490e4 <unknown>
#7 0x558a5cb4a0ae <unknown>
#8 0x558a5ce0cce1 <unknown>
#9 0x558a5ce10b7e <unknown>
#10 0x558a5cdfa4b5 <unknown>
#11 0x558a5ce117d6 <unknown>
#12 0x558a5cddddbf <unknown>
#13 0x558a5ce34748 <unknown>
#14 0x558a5ce34917 <unknown>
#15 0x558a5ce45773 <unknown>
#16 0x7f917e7cafa3 start_thread

        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at org.openqa.selenium.remote.W3CHandshakeResponse.lambda$errorHandler$0(W3CHandshakeResponse.java:62)
        at org.openqa.selenium.remote.HandshakeResponse.lambda$getResponseFunction$0(HandshakeResponse.java:30)
        at org.openqa.selenium.remote.ProtocolHandshake.lambda$createSession$0(ProtocolHandshake.java:126)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
        at java.base/java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
        at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
        at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:543)
        at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:128)
        at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:74)
        at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:136)
        at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
        at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)
        at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:213)
        at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:131)
        at ca.gc.cyber.kangooroo.browser.chrome.CustomChromeDriver.<init>(CustomChromeDriver.java:40)
        at ca.gc.cyber.kangooroo.browser.KangoorooChromeBrowser.createDriver(KangoorooChromeBrowser.java:527)
        at ca.gc.cyber.kangooroo.browser.KangoorooChromeBrowser.execute(KangoorooChromeBrowser.java:102)
        at ca.gc.cyber.kangooroo.browser.KangoorooBrowser.get(KangoorooBrowser.java:79)
        at ca.gc.cyber.kangooroo.KangoorooStandaloneRunner.main(KangoorooStandaloneRunner.java:281)

After those stack traces, Kangooroo runs further, but does nothing. However, I've managed to find some logs in the working tmp directory (...)/tmp/99999ebcfdb78df077ad2727fd00969f/Default/chrome_debug.log:

[1124/150902.132539:FATAL:zygote_host_impl_linux.cc(201)] Check failed: . : Operation not permitted (1)

This has finally directed me to this post, and indeed:

assemblyline@f11c03ceabb2:/opt/al_service/kangooroo$ google-chrome
Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
[1158:1158:1124/151604.877773:FATAL:zygote_host_impl_linux.cc(201)] Check failed: . : Operation not permitted (1)
Trace/breakpoint trap (core dumped)

I did this debugging locally, building the image from source (commit 9137c7434dbd3fefcdfa71f82e4611e1f83a7ce9); when I try to run the google-chrome command in the production stable28 image (set up by the AL), the results are similar:

assemblyline@98c8eed440c3:/opt/al_service$ google-chrome --headless 
Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
[1124/153249.616578:FATAL:zygote_host_impl_linux.cc(201)] Check failed: . : Operation not permitted (1)
[1124/153249.621367:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2)
[1124/153249.621437:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2)
[1124/153249.623460:ERROR:directory_reader_posix.cc(42)] opendir /tmp/Crashpad/attachments/772d2db5-70c8-47b1-a528-9ed669cfc6e7: No such file or directory (2)
Trace/breakpoint trap
assemblyline@98c8eed440c3:/opt/al_service$ [0100/000000.641883:ERROR:zygote_linux.cc(662)] write: Broken pipe (32)

(with or without --headless is the same)

It looks like in both, locally built and the downloaded image, I have the same Chrome version:

google-chrome --version
Google Chrome 119.0.6045.105

To Reproduce
Steps to reproduce the behavior:

  1. Install the stable28 downloader version
  2. Submit a URL
  3. Wait until failed.

Expected behavior
URLs are downloaded. In case of the error, appropriate logs are emitted: currently, without manual running, nothing helpful is preserved or logged.

Screenshots

Environment (please complete the following information if pertinent):

  • Assemblyline Version: 4.4.0.stable79
  • URLDownloader versions:
    • 4.4.0.stable17, 4.4.0.stable20 - empty results
    • 4.4.0.stable28, latest (from repo) - different errors
  • Browser: [e.g. chrome, safari]

Additional context
I'm not sure if the sandboxing is really the problem, or it's something else. But this is the only one clue I've found.
@kam193 kam193 added assess We still haven't decided if this will be worked on or not bug Something isn't working labels Nov 24, 2023
@gdesmar
Copy link

gdesmar commented Nov 24, 2023

Just to confirm, are you using a docker-compose setup? I believe it works on Kubernetes but not on docker-compose.

@kam193
Copy link
Author

kam193 commented Nov 24, 2023

Yes, I've forgotten to add this. I use the docker-compose.

@gdesmar
Copy link

gdesmar commented Nov 24, 2023

Sadly the code of Kangooroo is not (yet) open-sourced, but I can confirm that they are using the --headless and the --disable-dev-shm-usage options, which are two of the three important switches that I ended up using experimenting with Selenium, --no-sandbox being the last. I've raised the issue with the Kangooroo team and also pointed them toward the --headless=new option in case.

@gdesmar gdesmar added accepted This issue was accepted, we will work on this at some point service-urldownloader Related to URLDownloader service and removed assess We still haven't decided if this will be worked on or not labels Nov 24, 2023
@gdesmar gdesmar changed the title URLDownloader doesn't work URLDownloader doesn't work on docker-compose Dec 5, 2023
@gdesmar
Copy link

gdesmar commented Dec 5, 2023 •
edited

I got a new version of Kangooroo, which now allows setting the --no-sandbox option. I just created tag v4.4.1.dev64 and v4.4.0.stable31 which should add the no_sandbox option in the service parameters. It will default to False as it is not needed for Kubernetes deployments, but you can switch it to True manually. I'll add some documentation in URLDownloader's README.

For future notes, it was brought to my attention that it is recommended to use a seccomp profile when running the container instead of modifying the chrome runtime with --no-sandbox. Seccomp is like an allow list of syscalls, so this list should be what we need to run chrome. An example of a valid profile to use can be found at: https://raw.githubusercontent.com/jfrazelle/dotfiles/master/etc/docker/seccomp/chrome.json
I tested it (--security-opt seccomp=chrome.json) and was able to run without the --no-sandbox option. It may be worth investigating how a module can configure how it should be scaled up, but since it is specific for a docker-controller and not a kubernetes-controller, it may be complicated.

This should be enough to get google-chrome working on a docker deployment.
Feel free to reopen the ticket if you do not get the expected result! :)

@gdesmar gdesmar closed this as completed Dec 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
accepted This issue was accepted, we will work on this at some point bug Something isn't working service-urldownloader Related to URLDownloader service
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kam193 @gdesmar