Skip to content
This repository has been archived by the owner on Aug 2, 2023. It is now read-only.

Database design and deploy for API user accounts and metering #20

Closed
achimnol opened this issue Nov 11, 2016 · 0 comments
Closed

Database design and deploy for API user accounts and metering #20

achimnol opened this issue Nov 11, 2016 · 0 comments
Assignees
@achimnol
Labels
feature
Projects
User Management and Billing
Milestone
0.8

Comments

@achimnol
Copy link
Member

Details in WIP.
@achimnol achimnol added this to the 0.6 milestone Nov 11, 2016
@achimnol achimnol self-assigned this Nov 11, 2016
achimnol added a commit that referenced this issue Nov 16, 2016
@achimnol
refs #20: Add DB schema and its initialization steps.
96c88ca 
 * To see how to manage database, run:
   `python -m sorna.gateway.models --help`.

   - You can combine `--drop-tables`, `--create-tables`, and
     `--populate-fixtures` to choose which operations to execute.

 * Also fixes some minor formatting issues pointed out by flake8.
achimnol added a commit that referenced this issue Nov 16, 2016
@achimnol
refs #20: Add indexes to DB schema.
bf360cd 
 * Also add server-side default to GUID field.
   Currently we cannot use SQLAlchemy's ORM because of aiopg,
   and without ORM we need to specify SQL-level default values
   when inserting a new row.
achimnol added a commit that referenced this issue Nov 20, 2016
@achimnol
refs #20: Implement authorization using DB data
b308887 
 * Replaces aiopg with asyncpg for better performance.
   (Actually we use SQLAlchemy as a SQL compiler without ORM,
   so it doesn't matter much how the DB API looks like.)

 * Adapts test cases to use the database as well.
   (using the default setting: sorna db granted to postgres/develove on
   localhost)
achimnol added a commit that referenced this issue Nov 20, 2016
@achimnol
refs #19, #20: Implement "get info" API.
260104e 
 * Adds stats-related fields to Kernel struct.
achimnol added a commit that referenced this issue Nov 23, 2016
@achimnol
refs #20: Update KeyPair stats for authorized access
d95e417
achimnol added a commit that referenced this issue Nov 23, 2016
@achimnol
refs #15, #20: Increment per-kernel query count upon get-info API.
e192721 
 * Also clean up imports in registry and refactor dict2kvlist as a
   sorna-common utility function.
achimnol added a commit that referenced this issue Nov 24, 2016
@achimnol
refs #20: Redesign resource limit credits.
efbf02d 
 * Tracking remaining resources instead of quota allow more flexible
   billing policy by renewing them or adding specific amounts.
achimnol added a commit that referenced this issue Dec 3, 2016
@achimnol
refs #15, #20: Implement concurrency limit.
8b177b3 
 * Requires database schema update.

 * TODO: Restore usage when agents are missing or restarted.
achimnol added a commit that referenced this issue Dec 4, 2016
@achimnol
refs #15, #20: Implement a simple event bus.
dafdec9 
 * It is a generalized event dispatcher for the gateway/manager server.
   Now the gateway can add appropriate handlers when asynchronous events
   such as kernel termination occur.  Like manager-agent interaction,
   this also uses aiozmq.rpc facilities.

 * Implemented concurrency usage check further.

   - Now it is decremented when the kernel is actually destroyed.

   - TODO: decrement as well when the agent instance is destroyed
           or failed to deliver heartbeats due to HW/network failure.
           We need to reviv Redis subscription in sorna.gateway
           and dispatch "instance_terminated" event when shadow
           keys expires.
achimnol added a commit that referenced this issue Dec 5, 2016
@achimnol
refs #15, #20: Correct concurrency limit checks with row-level locking.
f39c59d
achimnol added a commit that referenced this issue Dec 16, 2016
@achimnol
refs #20, #21: Refine gateway-agent synchronization
1cc9f4f 
 * Gateway now has a grace period when starting up to detect
   currently alive agents and synchronize the database.

   - It collects all received events without processing them.
     Then chooses and runs the latest ones for each instance.

 * We now use Python 3.6 async-generators.  The first case is
   InstanceRegistry.enumerate_instance() method.

 * InstanceRegistry's get_kernel/get_instance methods now have
   a variant to retrieve only a single field from the Redis hash.

 * InstanceRegistry's get_kernel method now has a batched variant.
achimnol added a commit that referenced this issue Dec 20, 2016
@achimnol
refs #20: Stabilize gateway restart scenario.
3d5d10d 
 * Ensure "forget_instance" upon instance_terminated event to prevent
   bogus heartbeat timeout detections.

 * Add missing file.
achimnol added a commit that referenced this issue Dec 21, 2016
@achimnol
refs #20: Implement bulk-sync mechanism.
7bcc665
achimnol added a commit that referenced this issue Dec 21, 2016
@achimnol
refs #15, #20: Update DB schema and now record Usage.
149543f 
 * asyncpg needs to support timezone-enabled datetime objects. :(
achimnol added a commit that referenced this issue Dec 22, 2016
@achimnol
refs #20: Stabilize agent lost scenario.
65262e3
achimnol added a commit that referenced this issue Dec 22, 2016
@achimnol
refs #20: Optionall receive kernel stats upon kernel_terminated event.
5a1350f
achimnol added a commit that referenced this issue Dec 27, 2016
@achimnol
refs #20: Fix a critical error that failed kernel launches counts for…
ec63efe 
… concurrency limit.
achimnol added a commit that referenced this issue Jan 6, 2017
@achimnol
refs #20: Adjust DB schema to work with our internal cloud account ma…
2fe3361 
…nager.
achimnol added a commit that referenced this issue Jan 11, 2017
@achimnol
refs #15, #20: Stabilize agent lost/revival scenarios.
031a700 
 * Various bug fixes.

   - Don't forget_instance() when agent is lost.
     Required to check revival and resetting.

   - Fix wrong return value checks for HGETALL redis API: it returns an
     empty dict instead of None when the key does not exist.

   - update_kernel_usage() and update_instance_usage() needs to
     be able to access "stale" kernel information.

 * Shorten the range of connection leases from the redis pool.

 * Improve code styles.

   - Use parenthesis to long-hanging lines if possible.

   - Use .execute() instead of .fetchval() when the SQL query is
     doing insertion or updates.

   - Use more f-strings for brevity.
@achimnol achimnol modified the milestones: 0.6, 0.8 Jan 19, 2017
@achimnol achimnol moved this from In Progress to Done in User Management and Billing Apr 27, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@achimnol achimnol

Labels
feature
Projects
No open projects
User Management and Billing
Done
Milestone
0.8
Development

No branches or pull requests
1 participant
@achimnol