The Analytics change breaks AppRTC test in WebRTC waterfall

[jiayliu]

webrtc/samples#467 (comment)

phoglund commented 6 hours ago
I reverted the analytics change for now.
When bringing up the server on the bots after the analytics change:

INFO 2015-02-04 09:56:16,612 devappserver2.py:557] Skipping SDK update check.
WARNING 2015-02-04 09:56:16,622 simple_search_stub.py:1002] Could not read search indexes from /tmp/appengine.apprtc.chrome-bot/search_indexes
INFO 2015-02-04 09:56:16,623 api_server.py:138] Starting API server at: http://localhost:58086
INFO 2015-02-04 09:56:16,634 dispatcher.py:164] Starting module "default" running at: http://localhost:9999
ERROR 2015-02-04 09:56:17,886 wsgi.py:219]
Traceback (most recent call last):
File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 196, in Handle
handler = config_handle.add_wsgi_middleware(self.LoadHandler())
File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 255, in _LoadHandler
handler = __import(path[0])
File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/apprtc.py", line 26, in
import analytics
File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/analytics.py", line 14, in
from oauth2client.client import SignedJwtAssertionCredentials
ImportError: cannot import name SignedJwtAssertionCredentials
INFO 2015-02-04 09:56:17,891 module.py:593] default: "GET /r/some_room?wshpp=localhost:8089&wstls=false&firefox_fake_device=1 HTTP/1.1" 500 -
ERROR 2015-02-04 09:56:17,919 wsgi.py:219]
Traceback (most recent call last):
File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 196, in Handle
handler = config_handle.add_wsgi_middleware(self.LoadHandler())
File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 255, in _LoadHandler
handler = __import(path[0])
File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/apprtc.py", line 26, in
import analytics
File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/analytics.py", line 14, in
from oauth2client.client import SignedJwtAssertionCredentials
ImportError: cannot import name SignedJwtAssertionCredentials
INFO 2015-02-04 09:56:17,923 module.py:593] default: "GET /favicon.ico HTTP/1.1" 500 -
BrowserTestBase signal handler received SIGTERM. Backtrace:
#0 0x000001645df9 base::debug::StackTrace::StackTrace()

Log from http://chromegw.corp.google.com/i/chromium.webrtc/builders/Linux%20Tester/builds/33276/steps/browser_tests/logs/stdio.
phoglund
Owner
phoglund commented 6 hours ago
Where are the oauth2client imports supposed to come from? I can't see it being added or depended upon in the analytics CLs. Are they a part of appengine? We need to figure out why the bots fail to import those when they bring up the server with dev_appserver.py.

(Also, next time you do a merge like this, try to get it right in the branch before merging to master, so I don't have to revert 5 commits with fixes piled on top of each other).
phoglund
Owner
phoglund commented 2 hours ago
Strangely enough, the test works when I run on my workstation. I tried deleting my google_appengine copy and webrtc-samples; it still succeeds. Now that is weird. My first thought was that this is due to differences against the bot's PYTHONPATH, but it succeeds.

After digging around a bit, I reckon the code is supposed to come from src/out/webrtc-samples/samples/web/content/apprtc/third_party/oauth2client/client.py. Looking at that code:

if HAS_CRYPTO:

PyOpenSSL and PyCrypto are not prerequisites for oauth2client, so if it is

missing then don't create the SignedJwtAssertionCredentials or the

verify_id_token() method.

class SignedJwtAssertionCredentials(AssertionCredentials):

So the problem is that we don't have PyOpenSSL and PyCrypto? It wouldn't surprise me if the bots' system python do not have those things. Donald, what needs to happen to make that work? Can you pull the crypto stuff into third_party as some kind of build step for AppRTC? As a general thing, we can't install stuff into the bots' system python, so the app needs to pull all its dependencies by itself.

[jiayliu]

It also caused a problem on dev servers:

E 09:46:23.591 <HttpError 404 when requesting https://www.googleapis.com/bigquery/v2/projects/apprtc/datasets/prod/tables/analytics/insertAll?alt=json returned "Not Found: Project apprtc">
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in call
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in call
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in call
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(_args, *_kwargs)
File "/base/data/home/apps/sapprtc/3.382002904972735408/join_page.py", line 171, in post
allow_room_creation = True)
File "/base/data/home/apps/sapprtc/3.382002904972735408/room.py", line 213, in add_client_to_room
host=request.host)
File "/base/data/home/apps/sapprtc/3.382002904972735408/analytics.py", line 110, in report_event
analytics.report_event(_args, *_kwargs)
File "/base/data/home/apps/sapprtc/3.382002904972735408/analytics.py", line 91, in report_event
body=obj).execute()
File "/base/data/home/apps/sapprtc/3.382002904972735408/third_party/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(_args, *_kwargs)
File "/base/data/home/apps/sapprtc/3.382002904972735408/third_party/apiclient/http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 404 when requesting https://www.googleapis.com/bigquery/v2/projects/apprtc/datasets/prod/tables/analytics/insertAll?alt=json returned "Not Found: Project apprtc">

I'm going to disable the call for now

[milkypostman]

The app.yaml file specifies that App Engine should pull in the dependencies. I don't understand why this is a problem. When you run the App Engine dev server locally it is supposed to behave exactly like the server. https://github.com/webrtc/apprtc/blob/master/src/app_engine/app.yaml#L33

@phoglund why would the bots seem to be ignoring this?

[chuckhays]

To run python tests on mac (this is in the apprtc-t branch, but also happens in master) I have to add the following to analytics_test.py:
import os
import sys
sys.path.append(os.path.join(os.path.dirname(file), 'third_party'))
sys.path.append(os.path.join(os.path.dirname(file), '../third_party'))

I also have to comment out
#import analytics in room.py

I also noticed in the apprtc-t branch, apprtc_test.py doesn't have the same changes as master that add analytics tests.
Master: https://github.com/webrtc/apprtc/blob/master/src/app_engine/apprtc_test.py
apprtc-t: https://github.com/webrtc/apprtc/blob/apprtc-t/src/app_engine/apprtc_test.py

[jiayliu]

The analytics related code in apprtc_test.py is moved to test_utilities.py.

On Wed, Feb 4, 2015 at 11:39 AM, Chuck Hays notifications@github.com
wrote:

To run python tests on mac (this is in the apprtc-t branch, but also
happens in master) I have to add the following to analytics_test.py:
import os
import sys
sys.path.append(os.path.join(os.path.dirname(file), 'third_party'))
sys.path.append(os.path.join(os.path.dirname(file), '../third_party'))

I also have to comment out
#import analytics in room.py

I also noticed in the apprtc-t branch, apprtc_test.py doesn't have the
same changes as master that add analytics tests.
Master:
https://github.com/webrtc/apprtc/blob/master/src/app_engine/apprtc_test.py
apprtc-t:
https://github.com/webrtc/apprtc/blob/apprtc-t/src/app_engine/apprtc_test.py

—
Reply to this email directly or view it on GitHub
#21 (comment).

Jiayang Liu

[jiayliu]

I've setup the bigquery tables and there is no server error when running on cloud now.
We still need to figure out how to fix the tests.

[jiayliu]

BTW, I have to use the project number instead of the project id when calling bigquery.tabledata().insertAll. Not sure how to get project number programmatically.

[jiayliu]

I have #29 to fix the test issues, including the browser test issue and the unit test issue on mac.

[jiayliu]

Fixed

[phoglund]

Donald: yeah, so it would appear it is dev_appserver that somehow provisions these things. It also appears that provision fails. The bots are running rather strictly controlled version of python, so perhaps dev_appserver is looking for something that doesn't exist or is compatible with that version of python. I'll commandeer a bot and see if I can figure something out.

[phoglund]

Hm, well. The python version doesn't exactly look ancient.
Bot/Workstation
Python 2.7.3/Python 2.7.6
SDK 1.8.3/SDK 1.8.3

I can repro with
python google_appengine/dev_appserver.py src/out/webrtc-samples/samples/web/content/apprtc/
curl http://localhost:8080

on the bot.

[phoglund]

Downloading the latest SDK on the bot did not help, so something else is going on.

[phoglund]

Damn it, this is hard to debug. Here's the real error anyway:

import analytics
File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/analytics.py", line 17, in
import oauth2client.crypt
File "/b/build/slave/Linux_Tester/build/google_appengine/oauth2client/crypt.py", line 255, in
raise ImportError('No encryption library found. Please install either '
ImportError: No encryption library found. Please install either PyOpenSSL, or PyCrypto 2.6 or later

[milkypostman]

@phoglund i wonder if there is something missing on the bots which prevents pycrypto from being installed by dev_appserver.py. I guess I'll have to try more things locally.

[phoglund]

Ah, I was confused by the fact that the oauth2client code isn't pulled from apprtc/third_party/oauth2client, it's pulled from the app engine SDK folder/oauth2client. Keeping digging...

[phoglund]

Ok, so

Traceback (most recent call last):
File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 196, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 255, in _LoadHandler
handler = import(path[0])
File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/apprtc.py", line 26, in
import analytics
File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/analytics.py", line 13, in
from oauth2client.appengine import AppAssertionCredentials
File "/b/build/slave/Linux_Tester/build/google_appengine/oauth2client/appengine.py", line 46, in
from oauth2client.client import AccessTokenRefreshError
File "/b/build/slave/Linux_Tester/build/google_appengine/oauth2client/client.py", line 43, in
from oauth2client import crypt
File "/b/build/slave/Linux_Tester/build/google_appengine/oauth2client/crypt.py", line 37, in
from OpenSSL import crypto
ImportError: No module named OpenSSL

which is really weird, because if I run the system python on the bot I can import it fine. Must be that dev_appserver either launches its own python or hacks sys.path.

[milkypostman]

Well it's weird cause app.yaml specifies SSL and pycrypto.

[phoglund]

Aha, this is where the system gets it on my workstation:

/usr/lib/python2.7/dist-packages/Crypto/init.pyc

So Crypto, whatever that is, or openssh needs to be installed on the system python for this to work with dev_appserver.

[phoglund]

I guess pycrypto isn't as self-contained as you would think, i.e. needs stuff on the system.

[phoglund]

Ha! The bot actually manages to import parts of the Crypto library, but it fails because Crypto.Signature doesn't exist in the bot's version of the package. The Crypto version installed in /usr/lib/python2.7/dist-packages/Crypto/ is new enough on our workstations but not on the bot.

Well, now we know that at least. I'll give some thought on how to solve this tomorrow. I suspect upgrading all the bots somehow is out of the question. There's so much stuff using Python on the bots that an upgrade of anything is a major and risky undertaking. Donald, would it be possible to disable analytics when running under dev_appserver? It may try to load SignedJwtAssertionCredentials, but then it needs to catch the import error and do something appropriate (like shut itself down and not actually do anything when the rest of code calls it). WDYT?

[phoglund]

...Or maybe I can put a more recent copy of the package earlier in the sys.path to force Python to look it up there. The sys.path looks like this when I run:

/usr/local/google/home/phoglund/dev/chrome/src/out/webrtc-samples/samples/web/content/apprtc
/usr/local/google/home/phoglund/dev/chrome/google_appengine
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/protorpc
/usr/local/google/home/phoglund/dev/chrome/google_appengine
/usr/lib/python2.7
/usr/lib/python2.7/lib-dynload
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/jinja2-2.6
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/ssl-2.7
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/pycrypto-2.6
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/markupsafe-0.15
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/setuptools-0.6c11
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/webapp2-2.3
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/webob-1.1.1
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/yaml-3.10

That could work actually. Well, consider the above comment, that's certainly less hackish.

[milkypostman]

It would be very easy to fix this problem and ignore it on the bots. we
simply can import 'oauth2client.appengine' and then use
'AppAssertionCredentials' later based on that import.

On Thu, Feb 5, 2015 at 8:15 AM, phoglund notifications@github.com wrote:

...Or maybe I can put a more recent copy of the package earlier in the
sys.path to force Python to look it up there. The sys.path looks like this
when I run:

/usr/local/google/home/phoglund/dev/chrome/src/out/webrtc-samples/samples/web/content/apprtc
/usr/local/google/home/phoglund/dev/chrome/google_appengine
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/protorpc
/usr/local/google/home/phoglund/dev/chrome/google_appengine
/usr/lib/python2.7
/usr/lib/python2.7/lib-dynload
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/jinja2-2.6
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/ssl-2.7

/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/pycrypto-2.6

/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/markupsafe-0.15

/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/setuptools-0.6c11
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/webapp2-2.3
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/webob-1.1.1
/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/yaml-3.10

That could work actually. Well, consider the above comment, that's
certainly less hackish.

—
Reply to this email directly or view it on GitHub
#21 (comment).