Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

change the LH code to support closed recruitment #628

Open
shankari opened this issue Mar 15, 2021 · 4 comments
Open

change the LH code to support closed recruitment #628

shankari opened this issue Mar 15, 2021 · 4 comments

Comments

@shankari
Copy link
Contributor

Changes:

  • Client
    • start with the em-base UI
    • accept only one valid QR code (the "secret") to move on from the embase screen
    • on accepting the QR code, start regular onboarding
    • create a random token specific to the user
    • send the secret + user token to the server for auth
  • Server
    • change the auth to check for the secret, and to use the token to validate the user
    • remove support for calls without any auth token. These typically return aggregate data.

This will:

  • ensure that we will not collect any user PII, and
  • close the deployment so that only users with a valid QR code will be able to participate
    • since we also check for the secret on the server, connections to the server that don't have the QR code will fail
    • exceptions are GET calls to retrieve static text and images for the landing page
@shankari
Copy link
Contributor Author

start with the em-base UI

  • copy over the UI files, and the splash code
  • starts but doesn't stay there, moves on to another state
  • due to startprefs code, which is linked from splash code and determines the next step
    • we want to keep this code since we don't always want to start with the emTripLog screen
    • normally, this is handled by the ionic deploy, but we are putting in a hack to avoid that now
    • so we need to add a new state (similar to INTRO_DONE) called (WAITING_FOR_SECRET).
    • if waiting for secret, we need to stay in the embase page, otherwise we move on to the standard checks

@shankari
Copy link
Contributor Author

accept only one valid QR code (the "secret") to move on from the embase screen

should we make the secret configurable via file or hardcode it in the code?
Going to hardcode it now for ease of use and to highlight the fact that this is a hack
over the long term, we want to switch to some kind of channel again

@shankari
Copy link
Contributor Author

send the secret + user token to the server for auth

We read the hardcoded token from the module directly, since we only reach the login screen if the scanned token matches the hardcoded one.

@shankari
Copy link
Contributor Author

remove support for calls without any auth token. These typically return aggregate data.

Fixing #408 as well while implementing this. The challenge is that in the current implementation, we check whether to return user or aggregate values by looking whether we have a user token or not. And on the phone, we skip the user token when making aggregate calls. We need to change both the server and the client in that case.

shankari added a commit to shankari/e-mission-server that referenced this issue Mar 17, 2021
@shankari
Support various auth policies for aggregate calls as well
55715c2 
This fixes
e-mission/e-mission-docs#408

It is also a partial fix for
e-mission/e-mission-docs#628

We support 3 basic policies:
- `no_auth`: full public access (backwards compatible behavior)
- `user_only`: access only to existing users (new functionality, consistent with e-mission/e-mission-docs#408)
- `never`: disable completely

Other sophisticated access control for certain users only is out of the scope at this time

Testing done:
- set the policy to `no_auth`
    - aggregate call works

    ```
    2021-03-16 16:25:32,859:DEBUG:123145663979520:START POST /result/metrics/timestamp
    2021-03-16 16:25:32,859:DEBUG:123145663979520:Aggregate call, checking {aggregate_call_support} policy
    2021-03-16 16:25:32,859:DEBUG:123145663979520:metric_list = ['duration', 'median_speed', 'count', 'distance']
    2021-03-16 16:25:32,859:DEBUG:123145663979520:['duration -> <function get_duration at 0x7ffe61347cb0>', 'median_speed -> <function get_median_speed at 0x7ffe61347d40>', 'count -> <function get_count at 0x7ffe61347b90>', 'distance -> <function get_distance at 0x7ffe61347c20>']
    2021-03-16 16:25:32,859:DEBUG:123145663979520:for user None, returning timeseries <emission.storage.timeseries.aggregate_timeseries.AggregateTimeSeries object at 0x7ffe8052d790>
    2021-03-16 16:25:32,867:DEBUG:123145663979520:END POST /result/metrics/timestamp  0.008590936660766602
    ```

    - user call works

    ```
    2021-03-16 16:25:32,866:DEBUG:123145669234688:START POST /result/metrics/timestamp
    2021-03-16 16:25:32,867:DEBUG:123145669234688:User specific call, returning UUID
    2021-03-16 16:25:32,868:DEBUG:123145669234688:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'>
    2021-03-16 16:25:32,868:DEBUG:123145669234688:Using the skip method to verify id token REPLACEMEkVVdF9rT of length 17
    2021-03-16 16:25:32,870:DEBUG:123145669234688:retUUID = cf8ccb7b-84d7-40e4-a726-7691e614b042
    2021-03-16 16:25:32,876:DEBUG:123145669234688:END POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.009974002838134766
    ```

- switch the policy to `user_only`

    - user call works

    ```
    2021-03-16 16:25:32,866:DEBUG:123145669234688:START POST /result/metrics/timestamp
    2021-03-16 16:25:32,867:DEBUG:123145669234688:User specific call, returning UUID
    2021-03-16 16:25:32,868:DEBUG:123145669234688:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'>
    2021-03-16 16:25:32,868:DEBUG:123145669234688:Using the skip method to verify id token REPLACEMEkVVdF9rT of length 17
    2021-03-16 16:25:32,870:DEBUG:123145669234688:retUUID = cf8ccb7b-84d7-40e4-a726-7691e614b042
    2021-03-16 16:25:32,876:DEBUG:123145669234688:END POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.009974002838134766
    ```

    - aggregate call fails

    ```
    2021-03-16 16:59:25,517:DEBUG:123145504403456:START POST /result/metrics/timestamp
    2021-03-16 16:59:25,517:DEBUG:123145504403456:Aggregate call, checking user_only policy
    2021-03-16 16:59:25,518:DEBUG:123145504403456:END POST /result/metrics/timestamp  0.00035881996154785156
    ```

    with error

    ```
    2021-03-16 16:58:42.465 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "ERROR:Error loading aggregate data, averages not available{"status":403,"url":"http://10.0.2.2:8080/result/metrics/timestamp","headers":{"date":"Tue, 16 Mar 2021 23:59:25 GMT","content-length":"761","server":"Cheroot/8.4.2","x-android-selected-protocol":"http/1.1","x-android-response-source":"NETWORK 403","x-android-received-millis":"1615939122220","x-android-sent-millis":"1615939122206","content-type":"text/html; charset=UTF-8"},"error":"\n    <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n    <html>\n        <head>\n            <title>Error: 403 Forbidden</title>\n            <style type=\"text/css\">\n              html {background-color: #eee; font-family: sans-serif;}\n              body {background-color: #fff; border: 1px solid #ddd;\n                    padding: 15px; margin: 15px;}\n              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n            </style>\n        </head>\n        <body>\n            <h1>Error: 403 Forbidden</h1>\n            <p>Sorry, the requested URL <tt>&e-mission#39;http://10.0.2.2:8080/result/metrics/timestamp&#039;</tt>\n               caused an error:</p>\n            <pre>aggregations only available to users</pre>\n        </body>\n    </html>\n"}", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    ```

- switch the policy to `never`, fails with error

    ```
    2021-03-16 17:13:20.422 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "Error loading aggregate data, averages not available{"status":404,"url":"http://10.0.2.2:8080/result/metrics/timestamp","headers":{"date":"Wed, 17 Mar 2021 00:14:03 GMT","content-length":"754","server":"Cheroot/8.4.2","x-android-selected-protocol":"http/1.1","x-android-response-source":"NETWORK 404","x-android-received-millis":"1615940000171","x-android-sent-millis":"1615940000159","content-type":"text/html; charset=UTF-8"},"error":"\n    <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n    <html>\n        <head>\n            <title>Error: 404 Not Found</title>\n            <style type=\"text/css\">\n              html {background-color: #eee; font-family: sans-serif;}\n              body {background-color: #fff; border: 1px solid #ddd;\n                    padding: 15px; margin: 15px;}\n              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n            </style>\n        </head>\n        <body>\n            <h1>Error: 404 Not Found</h1>\n            <p>Sorry, the requested URL <tt>&e-mission#39;http://10.0.2.2:8080/result/metrics/timestamp&#039;</tt>\n               caused an error:</p>\n            <pre>Aggregate calls not supported</pre>\n        </body>\n    </html>\n"}", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    ```

- switch the policy to an invalid valid, fails with error

    ```
    2021-03-16 17:14:25.561 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "ERROR:Error loading aggregate data, averages not available{"status":500,"url":"http://10.0.2.2:8080/result/metrics/timestamp","headers":{"date":"Wed, 17 Mar 2021 00:15:08 GMT","content-length":"1550","server":"Cheroot/8.4.2","x-android-selected-protocol":"http/1.1","x-android-response-source":"NETWORK 500","x-android-received-millis":"1615940065310","x-android-sent-millis":"1615940065297","content-type":"text/html; charset=UTF-8"},"error":"\n    <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n    <html>\n        <head>\n            <title>Error: 500 Internal Server Error</title>\n            <style type=\"text/css\">\n              html {background-color: #eee; font-family: sans-serif;}\n              body {background-color: #fff; border: 1px solid #ddd;\n                    padding: 15px; margin: 15px;}\n              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n            </style>\n        </head>\n        <body>\n            <h1>Error: 500 Internal Server Error</h1>\n            <p>Sorry, the requested URL <tt>&e-mission#39;http://10.0.2.2:8080/result/metrics/timestamp&#039;</tt>\n               caused an error:</p>\n            <pre>Internal Server Error</pre>\n              <h2>Exception:</h2>\n              <pre>KeyError(&e-mission#39;foobar&e-mission#39;)</pre>\n              <h2>Traceback:</h2>\n              <pre>Traceback (most recent call last):\n  File &quot;/Users/kshankar/e-mission/e-mission-server/emission/net/api/bottle.py&quot;, line 997, in _handle\n    out = route.call(**args)\n  File &quot;/Users/kshankar/e-mission/e-mission-server/emission/net/api/bottle.py&quot;, line 1998, in wrapper\n    rv = callback(*a, **ka)\n  File &quot;emission/net/api/cfc_webapp.py&quot;, line 466, in summarize_metrics\n    user_uuid = get_user_or_aggregate_auth(request)\n  File &quot;emission/net/api/cfc_webapp.py&quot;, line 621, in get_user_or_aggregate_auth\n    return aggregate_call_map[aggregate_call_support](request)\nKeyError: &e-mission#39;foobar&e-mission#39;\n</pre>\n        </body>\n    </html>\n"}", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    ```

- changed the phone code to send a user token for aggregate calls as well, worked

    ```
    2021-03-16 18:52:47,214:DEBUG:123145648730112:START POST /result/metrics/timestamp
    2021-03-16 18:52:47,214:DEBUG:123145648730112:Aggregate call, checking user_only policy
    2021-03-16 18:52:47,214:DEBUG:123145648730112:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'>
    2021-03-16 18:52:47,215:DEBUG:123145648730112:Using the skip method to verify id token REPLACEMEkVVdF9rT of length 17
    2021-03-16 18:52:47,216:DEBUG:123145648730112:retUUID = cf8ccb7b-84d7-40e4-a726-7691e614b042
    2021-03-16 18:52:47,223:DEBUG:123145648730112:END POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.009236335754394531
    ```
@shankari shankari mentioned this issue Mar 17, 2021
Merged
shankari added a commit to shankari/e-mission-phone that referenced this issue Mar 17, 2021
@shankari
Add support for aggregate auth on this phone as well
c967f47 
This fixes e-mission/e-mission-docs#408
and is a partial fix for
e-mission/e-mission-docs#628

Fix is fairly straightforward:
- introduce a `aggregate_call_auth` config option, similar to the server
- if is it set to `no_auth`, use the `cordova.plugin.http.sendRequest` as before
- if it is set to `user_only`, call
  `window.cordova.plugins.BEMServerComm.pushGetJSON` similar to the existing
  user calls

`window.cordova.plugins.BEMServerComm.pushGetJSON` returns the data from the
response instead of the response directly. So change the `no_auth` option to
also return the response data, and change all calling functions to access
`response.foo` instead of `response.data.foo`

Bonus fix: `finally` breaks on android 27 emulator, use `then().catch()`
instead.

Testing done:
- with a server configured for `no_auth`
    - client = `no_auth` works
- with a server configured for `user_only`
    - client = `no_auth` fails
    ```
    2021-03-16 16:58:42.465 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "ERROR:Error loading aggregate data, averages not available{"status":403,"url":"http://10.0.2.2:8080/result/metrics/timestamp","headers":{"date":"Tue, 16 Mar 2021 23:59:25 GMT","content-length":"761","server":"Cheroot/8.4.2","x-android-selected-protocol":"http/1.1","x-android-response-source":"NETWORK 403","x-android-received-millis":"1615939122220","x-android-sent-millis":"1615939122206","content-type":"text/html; charset=UTF-8"},"error":"\n    <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n    <html>\n        <head>\n            <title>Error: 403 Forbidden</title>\n            <style type=\"text/css\">\n              html {background-color: #eee; font-family: sans-serif;}\n              body {background-color: #fff; border: 1px solid #ddd;\n                    padding: 15px; margin: 15px;}\n              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n            </style>\n        </head>\n        <body>\n            <h1>Error: 403 Forbidden</h1>\n            <p>Sorry, the requested URL <tt>&#039;http://10.0.2.2:8080/result/metrics/timestamp&#039;</tt>\n               caused an error:</p>\n            <pre>aggregations only available to users</pre>\n        </body>\n    </html>\n"}", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    ```
    - client = `user_only` succeeds
    ```
    2021-03-16 18:52:47,214:DEBUG:123145648730112:START POST /result/metrics/timestamp
    2021-03-16 18:52:47,214:DEBUG:123145648730112:Aggregate call, checking user_only policy
    2021-03-16 18:52:47,214:DEBUG:123145648730112:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'>
    2021-03-16 18:52:47,215:DEBUG:123145648730112:Using the skip method to verify id token REPLACEMEkVVdF9rT of length 17
    2021-03-16 18:52:47,216:DEBUG:123145648730112:retUUID = cf8ccb7b-84d7-40e4-a726-7691e614b042
    2021-03-16 18:52:47,223:DEBUG:123145648730112:END POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.009236335754394531
    ```
@shankari shankari mentioned this issue Mar 17, 2021
Merged
shankari added a commit to shankari/e-mission-server that referenced this issue Mar 17, 2021
@shankari
Support a new "secret" auth method on the phone
434fa64 
This is a partial fix for issue
e-mission/e-mission-docs#628

The new method assumes that all tokens are prefixed by a shared secret between
the server and the client. This means that all tokens have two parts both of which are validated by the method.
- The second part is used to identify the user, randomly generating it on the
  phone ensures that users can only see their own data.
- The first shared part is configured into both the app and the server. It
  ensures that users can only connect to the server from authorized clients,
  which in this case, are smartphone apps that embed the secret.

+ bonus fix to generate the correct error (403 instead of 401) in case of auth errors

Testing done:
- configured the webserver to use the secret method
- configured one valid secret `"FOOBAR"`
    - auth failed on the server
    ```
    END 2021-03-16 11:33:43.257294 POST /result/metrics/timestamp  0.00736689567565918
    Traceback (most recent call last):
      File "emission/net/api/cfc_webapp.py", line 594, in getUUID
        retUUID = enaa.getUUID(request, auth_method, inHeader)
      File "/Users/kshankar/e-mission/e-mission-server/emission/net/auth/auth.py", line 67, in getUUID
        retUUID = getUUIDFromToken(authMethod, userToken)
      File "/Users/kshankar/e-mission/e-mission-server/emission/net/auth/auth.py", line 41, in getUUIDFromToken
        userEmail = AuthMethodFactory.getAuthMethod(authMethod).verifyUserToken(token)
      File "/Users/kshankar/e-mission/e-mission-server/emission/net/auth/secret.py", line 29, in verifyUserToken
        (token, len(self.client_secret_list)))
    ValueError: Invalid token REPLACEMEkVVdF9rT, not found in list of length 1
    ```
    - reflected on the client
    ```
    2021-03-16 11:32:58.087 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "Error loading user data"While pushing/getting from server HTTP/1.1 403 Forbidden"", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    2021-03-16 11:32:58.087 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "ERROR:Error loading user data"While pushing/getting from server HTTP/1.1 403 Forbidden"", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    ```

- configured the same valid secret as the phone `"REPLACEME"`
    e-mission/e-mission-phone@c00f63b
    - auth succeded on server
    ```
    START 2021-03-16 11:36:22.818484 POST /result/metrics/timestamp
    END 2021-03-16 11:36:22.832453 POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.013911962509155273
    ```
    - reflected on client
    ```
    2021-03-16 11:36:21.821 23394-23539/edu.berkeley.eecs.emission.devapp D/ConnectionSettings: in getConnectURL, connectionSettings = {"connectUrl":"http:\/\/10.0.2.2:8080","android":{"auth":{"method":"prompted-auth","clientID":"ignored"}},"ios":{"auth":{"method":"prompted-auth","clientID":"ignored"}}}
    2021-03-16 11:36:21.832 23394-23539/edu.berkeley.eecs.emission.devapp D/ConnectionSettings: in getConnectURL, returning http://10.0.2.2:8080
    2021-03-16 11:36:21.836 23394-9405/edu.berkeley.eecs.emission.devapp I/System.out: Posting data to http://10.0.2.2:8080/result/metrics/timestamp
    2021-03-16 11:36:21.836 23394-9405/edu.berkeley.eecs.emission.devapp I/System.out: About to execute query SELECT data FROM userCache WHERE key = 'prompted-auth' AND type = 'local-storage' AND write_ts >= 0.0 AND write_ts <= 1.615919781836E12 ORDER BY write_ts DESC
    2021-03-16 11:36:21.848 23394-9405/edu.berkeley.eecs.emission.devapp I/PromptedAuth: Auth found in local storage, now it should be stable
    2021-03-16 11:36:21.887 23394-9405/edu.berkeley.eecs.emission.devapp I/CommunicationHelper: Got response org.apache.http.message.BasicHttpResponse@c89c5d6 with status HTTP/1.1 200 OK
    ```
shankari added a commit to shankari/e-mission-server that referenced this issue Mar 17, 2021
@shankari
Support a new "secret" auth method on the phone
460b9e4 
This is a partial fix for issue
e-mission/e-mission-docs#628

The new method assumes that all tokens are prefixed by a shared secret between
the server and the client. This means that all tokens have two parts both of which are validated by the method.
- The second part is used to identify the user, randomly generating it on the
  phone ensures that users can only see their own data.
- The first shared part is configured into both the app and the server. It
  ensures that users can only connect to the server from authorized clients,
  which in this case, are smartphone apps that embed the secret.

+ bonus fix to generate the correct error (403 instead of 401) in case of auth errors

Testing done:
- configured the webserver to use the secret method
- configured one valid secret `"FOOBAR"`
    - auth failed on the server
    ```
    END 2021-03-16 11:33:43.257294 POST /result/metrics/timestamp  0.00736689567565918
    Traceback (most recent call last):
      File "emission/net/api/cfc_webapp.py", line 594, in getUUID
        retUUID = enaa.getUUID(request, auth_method, inHeader)
      File "/Users/kshankar/e-mission/e-mission-server/emission/net/auth/auth.py", line 67, in getUUID
        retUUID = getUUIDFromToken(authMethod, userToken)
      File "/Users/kshankar/e-mission/e-mission-server/emission/net/auth/auth.py", line 41, in getUUIDFromToken
        userEmail = AuthMethodFactory.getAuthMethod(authMethod).verifyUserToken(token)
      File "/Users/kshankar/e-mission/e-mission-server/emission/net/auth/secret.py", line 29, in verifyUserToken
        (token, len(self.client_secret_list)))
    ValueError: Invalid token REPLACEMEkVVdF9rT, not found in list of length 1
    ```
    - reflected on the client
    ```
    2021-03-16 11:32:58.087 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "Error loading user data"While pushing/getting from server HTTP/1.1 403 Forbidden"", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    2021-03-16 11:32:58.087 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "ERROR:Error loading user data"While pushing/getting from server HTTP/1.1 403 Forbidden"", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    ```

- configured the same valid secret as the phone `"REPLACEME"`
    e-mission/e-mission-phone@c00f63b
    - auth succeded on server
    ```
    START 2021-03-16 11:36:22.818484 POST /result/metrics/timestamp
    END 2021-03-16 11:36:22.832453 POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.013911962509155273
    ```
    - reflected on client
    ```
    2021-03-16 11:36:21.821 23394-23539/edu.berkeley.eecs.emission.devapp D/ConnectionSettings: in getConnectURL, connectionSettings = {"connectUrl":"http:\/\/10.0.2.2:8080","android":{"auth":{"method":"prompted-auth","clientID":"ignored"}},"ios":{"auth":{"method":"prompted-auth","clientID":"ignored"}}}
    2021-03-16 11:36:21.832 23394-23539/edu.berkeley.eecs.emission.devapp D/ConnectionSettings: in getConnectURL, returning http://10.0.2.2:8080
    2021-03-16 11:36:21.836 23394-9405/edu.berkeley.eecs.emission.devapp I/System.out: Posting data to http://10.0.2.2:8080/result/metrics/timestamp
    2021-03-16 11:36:21.836 23394-9405/edu.berkeley.eecs.emission.devapp I/System.out: About to execute query SELECT data FROM userCache WHERE key = 'prompted-auth' AND type = 'local-storage' AND write_ts >= 0.0 AND write_ts <= 1.615919781836E12 ORDER BY write_ts DESC
    2021-03-16 11:36:21.848 23394-9405/edu.berkeley.eecs.emission.devapp I/PromptedAuth: Auth found in local storage, now it should be stable
    2021-03-16 11:36:21.887 23394-9405/edu.berkeley.eecs.emission.devapp I/CommunicationHelper: Got response org.apache.http.message.BasicHttpResponse@c89c5d6 with status HTTP/1.1 200 OK
    ```
@shankari shankari mentioned this issue Mar 17, 2021
Merged
shankari added a commit to shankari/e-mission-phone that referenced this issue Mar 17, 2021
@shankari
Add a secret check before proceeding
c00f63b 
This is a hack to support the specific requirements of the NREL publication
process, which is currently intended to be a closed system.
- Users need to scan a specified QR code that represents a secret
- They are allowed to proceed only if the secret matches the app secret

In the long term, we want to use channels directly instead of this "secret"
code. However, we didn't want to deal with externally hosted software for
approval at this time.

This is the commit fixes the first three tasks in
e-mission/e-mission-docs#628

in particular,
e-mission/e-mission-docs#628 (comment)
and
e-mission/e-mission-docs#628 (comment)

Changes:
- new "secretcheck" service
- on receiving a custom URL, pass the parameter to secretcheck
- secretcheck validates the secret and stores it
- change startprefs so that if we have a valid stored secret, we can move on to
  checking the other onboarding states
- copy over the templates and the javascript code for the first screen from the
  em-base repository
shankari added a commit to shankari/e-mission-phone that referenced this issue Mar 17, 2021
@shankari
Include the secret into the user authentication
5afea5e 
Fairly simple fix
We read the hardcoded secret directly from the `SecretCheck` module since we
should not have come to this screen unless the input `SecretCheck` and the
hardcoded secret match, so we might as well use the hardcoded check.

This is the final set of changes for
e-mission/e-mission-docs#628

Testing done:

Checked the userid on the server.

```
 {'_id': ObjectId('60503a011fa22fb602d332b1'),
  'user_email': 'REPLACEMEkVVdF9rT',
  'uuid': UUID('cf8ccb7b-84d7-40e4-a726-7691e614b042'),
  'update_ts': datetime.datetime(2021, 3, 15, 21, 54, 25, 463000)}]
```

Checked the userid on the phone - was the same

Checked the server logs, auth was fine

```
START 2021-03-15 21:54:25.460376 POST /profile/create
END 2021-03-15 21:54:25.467684 POST /profile/create  0.007261991500854492
START 2021-03-15 21:54:25.583845 POST /profile/get
END 2021-03-15 21:54:25.589675 POST /profile/get cf8ccb7b-84d7-40e4-a726-7691e614b042 0.005787849426269531
START 2021-03-15 21:54:26.051814 POST /profile/update
END 2021-03-15 21:54:26.057612 POST /profile/update cf8ccb7b-84d7-40e4-a726-7691e614b042 0.0057489871978759766
START 2021-03-15 21:55:32.222428 POST /result/metrics/timestamp
END 2021-03-15 21:55:32.234051 POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.011476993560791016
```
@shankari shankari mentioned this issue Mar 17, 2021
Merged
@shankari shankari mentioned this issue May 17, 2021
Merged
jf87 pushed a commit to jf87/e-mission-server that referenced this issue Jun 21, 2021
@shankari
Support various auth policies for aggregate calls as well
2889172 
This fixes
e-mission/e-mission-docs#408

It is also a partial fix for
e-mission/e-mission-docs#628

We support 3 basic policies:
- `no_auth`: full public access (backwards compatible behavior)
- `user_only`: access only to existing users (new functionality, consistent with e-mission/e-mission-docs#408)
- `never`: disable completely

Other sophisticated access control for certain users only is out of the scope at this time

Testing done:
- set the policy to `no_auth`
    - aggregate call works

    ```
    2021-03-16 16:25:32,859:DEBUG:123145663979520:START POST /result/metrics/timestamp
    2021-03-16 16:25:32,859:DEBUG:123145663979520:Aggregate call, checking {aggregate_call_support} policy
    2021-03-16 16:25:32,859:DEBUG:123145663979520:metric_list = ['duration', 'median_speed', 'count', 'distance']
    2021-03-16 16:25:32,859:DEBUG:123145663979520:['duration -> <function get_duration at 0x7ffe61347cb0>', 'median_speed -> <function get_median_speed at 0x7ffe61347d40>', 'count -> <function get_count at 0x7ffe61347b90>', 'distance -> <function get_distance at 0x7ffe61347c20>']
    2021-03-16 16:25:32,859:DEBUG:123145663979520:for user None, returning timeseries <emission.storage.timeseries.aggregate_timeseries.AggregateTimeSeries object at 0x7ffe8052d790>
    2021-03-16 16:25:32,867:DEBUG:123145663979520:END POST /result/metrics/timestamp  0.008590936660766602
    ```

    - user call works

    ```
    2021-03-16 16:25:32,866:DEBUG:123145669234688:START POST /result/metrics/timestamp
    2021-03-16 16:25:32,867:DEBUG:123145669234688:User specific call, returning UUID
    2021-03-16 16:25:32,868:DEBUG:123145669234688:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'>
    2021-03-16 16:25:32,868:DEBUG:123145669234688:Using the skip method to verify id token REPLACEMEkVVdF9rT of length 17
    2021-03-16 16:25:32,870:DEBUG:123145669234688:retUUID = cf8ccb7b-84d7-40e4-a726-7691e614b042
    2021-03-16 16:25:32,876:DEBUG:123145669234688:END POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.009974002838134766
    ```

- switch the policy to `user_only`

    - user call works

    ```
    2021-03-16 16:25:32,866:DEBUG:123145669234688:START POST /result/metrics/timestamp
    2021-03-16 16:25:32,867:DEBUG:123145669234688:User specific call, returning UUID
    2021-03-16 16:25:32,868:DEBUG:123145669234688:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'>
    2021-03-16 16:25:32,868:DEBUG:123145669234688:Using the skip method to verify id token REPLACEMEkVVdF9rT of length 17
    2021-03-16 16:25:32,870:DEBUG:123145669234688:retUUID = cf8ccb7b-84d7-40e4-a726-7691e614b042
    2021-03-16 16:25:32,876:DEBUG:123145669234688:END POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.009974002838134766
    ```

    - aggregate call fails

    ```
    2021-03-16 16:59:25,517:DEBUG:123145504403456:START POST /result/metrics/timestamp
    2021-03-16 16:59:25,517:DEBUG:123145504403456:Aggregate call, checking user_only policy
    2021-03-16 16:59:25,518:DEBUG:123145504403456:END POST /result/metrics/timestamp  0.00035881996154785156
    ```

    with error

    ```
    2021-03-16 16:58:42.465 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "ERROR:Error loading aggregate data, averages not available{"status":403,"url":"http://10.0.2.2:8080/result/metrics/timestamp","headers":{"date":"Tue, 16 Mar 2021 23:59:25 GMT","content-length":"761","server":"Cheroot/8.4.2","x-android-selected-protocol":"http/1.1","x-android-response-source":"NETWORK 403","x-android-received-millis":"1615939122220","x-android-sent-millis":"1615939122206","content-type":"text/html; charset=UTF-8"},"error":"\n    <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n    <html>\n        <head>\n            <title>Error: 403 Forbidden</title>\n            <style type=\"text/css\">\n              html {background-color: #eee; font-family: sans-serif;}\n              body {background-color: #fff; border: 1px solid #ddd;\n                    padding: 15px; margin: 15px;}\n              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n            </style>\n        </head>\n        <body>\n            <h1>Error: 403 Forbidden</h1>\n            <p>Sorry, the requested URL <tt>&e-mission#39;http://10.0.2.2:8080/result/metrics/timestamp&#039;</tt>\n               caused an error:</p>\n            <pre>aggregations only available to users</pre>\n        </body>\n    </html>\n"}", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    ```

- switch the policy to `never`, fails with error

    ```
    2021-03-16 17:13:20.422 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "Error loading aggregate data, averages not available{"status":404,"url":"http://10.0.2.2:8080/result/metrics/timestamp","headers":{"date":"Wed, 17 Mar 2021 00:14:03 GMT","content-length":"754","server":"Cheroot/8.4.2","x-android-selected-protocol":"http/1.1","x-android-response-source":"NETWORK 404","x-android-received-millis":"1615940000171","x-android-sent-millis":"1615940000159","content-type":"text/html; charset=UTF-8"},"error":"\n    <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n    <html>\n        <head>\n            <title>Error: 404 Not Found</title>\n            <style type=\"text/css\">\n              html {background-color: #eee; font-family: sans-serif;}\n              body {background-color: #fff; border: 1px solid #ddd;\n                    padding: 15px; margin: 15px;}\n              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n            </style>\n        </head>\n        <body>\n            <h1>Error: 404 Not Found</h1>\n            <p>Sorry, the requested URL <tt>&e-mission#39;http://10.0.2.2:8080/result/metrics/timestamp&#039;</tt>\n               caused an error:</p>\n            <pre>Aggregate calls not supported</pre>\n        </body>\n    </html>\n"}", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    ```

- switch the policy to an invalid valid, fails with error

    ```
    2021-03-16 17:14:25.561 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "ERROR:Error loading aggregate data, averages not available{"status":500,"url":"http://10.0.2.2:8080/result/metrics/timestamp","headers":{"date":"Wed, 17 Mar 2021 00:15:08 GMT","content-length":"1550","server":"Cheroot/8.4.2","x-android-selected-protocol":"http/1.1","x-android-response-source":"NETWORK 500","x-android-received-millis":"1615940065310","x-android-sent-millis":"1615940065297","content-type":"text/html; charset=UTF-8"},"error":"\n    <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n    <html>\n        <head>\n            <title>Error: 500 Internal Server Error</title>\n            <style type=\"text/css\">\n              html {background-color: #eee; font-family: sans-serif;}\n              body {background-color: #fff; border: 1px solid #ddd;\n                    padding: 15px; margin: 15px;}\n              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n            </style>\n        </head>\n        <body>\n            <h1>Error: 500 Internal Server Error</h1>\n            <p>Sorry, the requested URL <tt>&e-mission#39;http://10.0.2.2:8080/result/metrics/timestamp&#039;</tt>\n               caused an error:</p>\n            <pre>Internal Server Error</pre>\n              <h2>Exception:</h2>\n              <pre>KeyError(&e-mission#39;foobar&e-mission#39;)</pre>\n              <h2>Traceback:</h2>\n              <pre>Traceback (most recent call last):\n  File &quot;/Users/kshankar/e-mission/e-mission-server/emission/net/api/bottle.py&quot;, line 997, in _handle\n    out = route.call(**args)\n  File &quot;/Users/kshankar/e-mission/e-mission-server/emission/net/api/bottle.py&quot;, line 1998, in wrapper\n    rv = callback(*a, **ka)\n  File &quot;emission/net/api/cfc_webapp.py&quot;, line 466, in summarize_metrics\n    user_uuid = get_user_or_aggregate_auth(request)\n  File &quot;emission/net/api/cfc_webapp.py&quot;, line 621, in get_user_or_aggregate_auth\n    return aggregate_call_map[aggregate_call_support](request)\nKeyError: &e-mission#39;foobar&e-mission#39;\n</pre>\n        </body>\n    </html>\n"}", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    ```

- changed the phone code to send a user token for aggregate calls as well, worked

    ```
    2021-03-16 18:52:47,214:DEBUG:123145648730112:START POST /result/metrics/timestamp
    2021-03-16 18:52:47,214:DEBUG:123145648730112:Aggregate call, checking user_only policy
    2021-03-16 18:52:47,214:DEBUG:123145648730112:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'>
    2021-03-16 18:52:47,215:DEBUG:123145648730112:Using the skip method to verify id token REPLACEMEkVVdF9rT of length 17
    2021-03-16 18:52:47,216:DEBUG:123145648730112:retUUID = cf8ccb7b-84d7-40e4-a726-7691e614b042
    2021-03-16 18:52:47,223:DEBUG:123145648730112:END POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.009236335754394531
    ```
jf87 pushed a commit to jf87/e-mission-server that referenced this issue Jun 21, 2021
@shankari
Support a new "secret" auth method on the phone
5bf98b5 
This is a partial fix for issue
e-mission/e-mission-docs#628

The new method assumes that all tokens are prefixed by a shared secret between
the server and the client. This means that all tokens have two parts both of which are validated by the method.
- The second part is used to identify the user, randomly generating it on the
  phone ensures that users can only see their own data.
- The first shared part is configured into both the app and the server. It
  ensures that users can only connect to the server from authorized clients,
  which in this case, are smartphone apps that embed the secret.

+ bonus fix to generate the correct error (403 instead of 401) in case of auth errors

Testing done:
- configured the webserver to use the secret method
- configured one valid secret `"FOOBAR"`
    - auth failed on the server
    ```
    END 2021-03-16 11:33:43.257294 POST /result/metrics/timestamp  0.00736689567565918
    Traceback (most recent call last):
      File "emission/net/api/cfc_webapp.py", line 594, in getUUID
        retUUID = enaa.getUUID(request, auth_method, inHeader)
      File "/Users/kshankar/e-mission/e-mission-server/emission/net/auth/auth.py", line 67, in getUUID
        retUUID = getUUIDFromToken(authMethod, userToken)
      File "/Users/kshankar/e-mission/e-mission-server/emission/net/auth/auth.py", line 41, in getUUIDFromToken
        userEmail = AuthMethodFactory.getAuthMethod(authMethod).verifyUserToken(token)
      File "/Users/kshankar/e-mission/e-mission-server/emission/net/auth/secret.py", line 29, in verifyUserToken
        (token, len(self.client_secret_list)))
    ValueError: Invalid token REPLACEMEkVVdF9rT, not found in list of length 1
    ```
    - reflected on the client
    ```
    2021-03-16 11:32:58.087 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "Error loading user data"While pushing/getting from server HTTP/1.1 403 Forbidden"", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    2021-03-16 11:32:58.087 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "ERROR:Error loading user data"While pushing/getting from server HTTP/1.1 403 Forbidden"", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145)
    ```

- configured the same valid secret as the phone `"REPLACEME"`
    e-mission/e-mission-phone@c00f63b
    - auth succeded on server
    ```
    START 2021-03-16 11:36:22.818484 POST /result/metrics/timestamp
    END 2021-03-16 11:36:22.832453 POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.013911962509155273
    ```
    - reflected on client
    ```
    2021-03-16 11:36:21.821 23394-23539/edu.berkeley.eecs.emission.devapp D/ConnectionSettings: in getConnectURL, connectionSettings = {"connectUrl":"http:\/\/10.0.2.2:8080","android":{"auth":{"method":"prompted-auth","clientID":"ignored"}},"ios":{"auth":{"method":"prompted-auth","clientID":"ignored"}}}
    2021-03-16 11:36:21.832 23394-23539/edu.berkeley.eecs.emission.devapp D/ConnectionSettings: in getConnectURL, returning http://10.0.2.2:8080
    2021-03-16 11:36:21.836 23394-9405/edu.berkeley.eecs.emission.devapp I/System.out: Posting data to http://10.0.2.2:8080/result/metrics/timestamp
    2021-03-16 11:36:21.836 23394-9405/edu.berkeley.eecs.emission.devapp I/System.out: About to execute query SELECT data FROM userCache WHERE key = 'prompted-auth' AND type = 'local-storage' AND write_ts >= 0.0 AND write_ts <= 1.615919781836E12 ORDER BY write_ts DESC
    2021-03-16 11:36:21.848 23394-9405/edu.berkeley.eecs.emission.devapp I/PromptedAuth: Auth found in local storage, now it should be stable
    2021-03-16 11:36:21.887 23394-9405/edu.berkeley.eecs.emission.devapp I/CommunicationHelper: Got response org.apache.http.message.BasicHttpResponse@c89c5d6 with status HTTP/1.1 200 OK
    ```
@MukuFlash03 MukuFlash03 mentioned this issue Mar 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@shankari