Skip to content
This repository has been archived by the owner on Aug 2, 2023. It is now read-only.

REST API and Documentation for APIs #15

Closed
6 of 7 tasks
achimnol opened this issue Mar 1, 2016 · 1 comment
Closed
6 of 7 tasks

REST API and Documentation for APIs #15

achimnol opened this issue Mar 1, 2016 · 1 comment
Labels
feature

Comments

@achimnol
Copy link
Member

achimnol commented Mar 1, 2016 •
edited
Loading

Let's add an API to Sorna so that external services can easily integrate with this code execution service.

  • API Design
    • Authentication like AWS API
    • Standardized error response (candidate: RFC-7807)
    • Kernel/session management
    • Code execution and result returns in various forms (including WebSocket-based stdout streaming and customized display data)
  • Backend
    • Usage metrics and statistics
    • Billing
  • Documentation
    • Create a readthedocs site.
achimnol added a commit that referenced this issue Aug 30, 2016
@achimnol
refs #15: Add skeleton for API gateway server.
70c2289
achimnol added a commit that referenced this issue Aug 30, 2016
@achimnol
refs #1, #15: Add test suite bases for API gateway.
592f7cf 
 * It uses pytest. Run "pytest sorna" in the working copy root to
   execute all test cases.
achimnol added a commit that referenced this issue Sep 2, 2016
@achimnol
refs #1, #15: Add tests for checking API query timestamp.
0dad092
achimnol added a commit that referenced this issue Sep 19, 2016
@achimnol
refs #1, #15: Update codes/tests for finalized auth API spec.
3c5cda8
achimnol added a commit that referenced this issue Sep 20, 2016
@achimnol
refs #1, #15: Pass tests with different HMAC hashing schemes
611cff7
achimnol added a commit that referenced this issue Sep 20, 2016
@achimnol
refs #1, #15: Fix wrong base64-encoding of signature.
03bfe38 
 * It should be just a plain hex-encoded string as the spec says.
achimnol added a commit that referenced this issue Oct 11, 2016
@achimnol
refs #15: First adaptation of legacy interface to RESTful API.
09d3e0c 
 * TODO: We need to refactor sorna-side session mgmt (user_id + entry_id)
   since this CodeOnWeb-specific session ID may not be applicable to
   other API clients.
achimnol added a commit that referenced this issue Nov 16, 2016
@achimnol
refs #15: Clean up.
8b260c6 
 * Implement graceful shutdown for the REST API server.

 * Separate main-specific config arguments.
achimnol added a commit that referenced this issue Nov 22, 2016
@achimnol
refs #15: Now kernel creation/getinfo/destruction API works.
220ff4c
achimnol added a commit that referenced this issue Nov 23, 2016
@achimnol
refs #15: Change the primary kernel session key to opaque client token
039ffb8 
 * Client-side session token is a generalized concept for the composite
   primary key to indentify a kernel: (user_id, entry_id, lang)
achimnol added a commit that referenced this issue Nov 23, 2016
@achimnol
refs #15, #20: Increment per-kernel query count upon get-info API.
e192721 
 * Also clean up imports in registry and refactor dict2kvlist as a
   sorna-common utility function.
achimnol added a commit that referenced this issue Nov 23, 2016
@achimnol
refs #15: Implement query-mode execution API.
4ebc1fd 
 * Also fortify handling of JSON decode errors with malformed
   request bodies.
achimnol added a commit that referenced this issue Nov 23, 2016
@achimnol
refs #15: Improve timeout handling and minor fixes.
c46785f
achimnol added a commit that referenced this issue Nov 24, 2016
@achimnol
refs #15: Improve handling of signatures and error cases.
139a0c0
achimnol added a commit that referenced this issue Nov 24, 2016
@achimnol
refs #15: Implement API query rate limiting.
191660d 
 * Also apply sorna.exceptions name refactoring.
achimnol added a commit that referenced this issue Nov 24, 2016
@achimnol
refs #15: Quantize time precision for rate limiting to msec.
f8b1ebd 
 * Checked that it is working as expected by client-side integration
   tests and printing response headers.
achimnol added a commit that referenced this issue Nov 24, 2016
@achimnol
refs #15: Fortify exception handling with new exception hierarchy.
ef84b4f 
 * Now it bubbles agent-side errors with full details (type & title)
   in RFC7807 style.
achimnol added a commit that referenced this issue Dec 3, 2016
@achimnol
refs #15, #20: Implement concurrency limit.
8b177b3 
 * Requires database schema update.

 * TODO: Restore usage when agents are missing or restarted.
achimnol added a commit that referenced this issue Dec 4, 2016
@achimnol
refs #15, #20: Implement a simple event bus.
dafdec9 
 * It is a generalized event dispatcher for the gateway/manager server.
   Now the gateway can add appropriate handlers when asynchronous events
   such as kernel termination occur.  Like manager-agent interaction,
   this also uses aiozmq.rpc facilities.

 * Implemented concurrency usage check further.

   - Now it is decremented when the kernel is actually destroyed.

   - TODO: decrement as well when the agent instance is destroyed
           or failed to deliver heartbeats due to HW/network failure.
           We need to reviv Redis subscription in sorna.gateway
           and dispatch "instance_terminated" event when shadow
           keys expires.
achimnol added a commit that referenced this issue Dec 5, 2016
@achimnol
refs #15, #20: Correct concurrency limit checks with row-level locking.
f39c59d
achimnol added a commit that referenced this issue Dec 20, 2016
@achimnol
refs #15: Add AWS-like keypair generator as a module command
ca71d97 
 * Run `python -m sorna.gateway.auth --generate-keypair` to create one.

 * For programmatic uses, use sorna.gateway.auth.generate_keypair
   function.
achimnol added a commit that referenced this issue Dec 20, 2016
@achimnol
refs #15: Use a secure string comparator to prevent timing attacks.
cfc3a51
achimnol added a commit that referenced this issue Dec 21, 2016
@achimnol
refs #15, #20: Update DB schema and now record Usage.
149543f 
 * asyncpg needs to support timezone-enabled datetime objects. :(
achimnol added a commit that referenced this issue Dec 23, 2016
@achimnol
refs #15: Implement streaming mode API for web terminals.
944fcc6 
 * Imported most functionality from an interal front-end proxy.

 * TODO: stabilization (unclean termination often makes the event loop
   to hang indefinitely without explicit errors.)
achimnol added a commit that referenced this issue Jan 11, 2017
@achimnol
refs #15, #20: Stabilize agent lost/revival scenarios.
031a700 
 * Various bug fixes.

   - Don't forget_instance() when agent is lost.
     Required to check revival and resetting.

   - Fix wrong return value checks for HGETALL redis API: it returns an
     empty dict instead of None when the key does not exist.

   - update_kernel_usage() and update_instance_usage() needs to
     be able to access "stale" kernel information.

 * Shorten the range of connection leases from the redis pool.

 * Improve code styles.

   - Use parenthesis to long-hanging lines if possible.

   - Use .execute() instead of .fetchval() when the SQL query is
     doing insertion or updates.

   - Use more f-strings for brevity.
@achimnol
Copy link
Member Author

Billing is now moved to internal project. Closing.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@achimnol