diff --git a/README.md b/README.md index ff1a7e9..f09c143 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ Using the declarative power from UtilMeta, you can easily write APIs with auto r UtilMeta developed a standard that support all major Python web framework like **django**, **flask**, **fastapi** (starlette), **sanic**, **tornado** as runtime backend, and support current projects using these frameworks to develop new API using UtilMeta progressively drawing ### Highly Flexible & Extensible -UtilMeta is highly flexible with a series of plugins includes authentication (Session/JWT), CORS, rate limit, retry, and can be extended to support more features. +UtilMeta is highly flexible with a series of plugins including authentication (Session/JWT), CORS, rate limit, retry, and can be extended to support more features. ### Full-lifecycle DevOps Solution The [UtilMeta Platform](https://utilmeta.com/) provided the full-lifecycle DevOps solution for this framework, the API Docs, Debug, Logs, Monitoring, Alerts, Analytics will all been taken care of in the platform @@ -126,7 +126,7 @@ if you request the ArticleAPI like `GET /article?id=1`, you will get the result "content": "hello world" } ``` -This is conform to what you declared, and the OpenAPI docs will be generated automatically +This conforms to what you declared, and the OpenAPI docs will be generated automatically ### Migrate Integrate current django/flask/fastapi/... project with UtilMeta API is as easy as follows diff --git a/docs/en/tutorials/user-auth.md b/docs/en/tutorials/user-auth.md index 47d05bc..8dd77c8 100644 --- a/docs/en/tutorials/user-auth.md +++ b/docs/en/tutorials/user-auth.md @@ -16,7 +16,7 @@ meta setup demo-user ``` Enter `django` when prompted to select backend -After the project is created, we need to configure the database connection of the service, open it `server.py`, and insert the following code +After the project is created, we need to configure the database connection of the service, open `server.py`, and insert the following code ```python service = UtilMeta(...) @@ -54,7 +54,7 @@ You can see that a new folder named `user` has been created in our project folde The `migrations` folder is where Django handles the database migrations of the models -Once the app is created, we insert a line into the Django settings of `server.py` to specify the app. +Once the app is created, we insert a line into the `DjangoSettings` of `server.py` to specify the app. ```python hl_lines="3" service.use(DjangoSettings( @@ -67,7 +67,7 @@ So far, we have completed the configuration and initialization of the project. ## 2. Write user model -The user’s login registration API, of course, revolves around the “user”. Before developing the API, we first write the user’s data model. We open `user/models.py` and write +The user APIs depends on the "**user**", so before developing the API, we should write the user’s data model. We open `user/models.py` and write ```python from django.db import models from utilmeta.core.orm.backends.django.models import AbstractSession, PasswordField @@ -146,11 +146,11 @@ user_config = auth.User( In this code, `SessionSchema` is the core engine that processes and stores Session data, `session_config` declares the Session configuration with Session model and engine we just wrote, and configures the corresponding Cookie policy !!! tip - We use session store based on database to simply our tutorial, in practive, we often use cache+db as the store, you can find more in [Session Authentication](../../guide/auth#session) + We use session store based on database to simply our tutorial, in practice, we often use cache+db as the store, you can find more in [Session Authentication](../../guide/auth#session) We also declare the user authentication configuration `user_config` with the following params -* `user_model` Specify the user model for authentication, which is the User model I wrote in the previous section. +* `user_model`: Specify the user model for authentication, which is the User model I wrote in the previous section. * `authentication`: Specify the authentication method. We pass `session_config` in to declare that user authentication is performed using Session. * `key`: Specify the key of the current user ID in the session data * `login_fields`: Fields that can be used for login, such as username, email, etc., which need to be unique. @@ -203,7 +203,7 @@ The logic in the signup API function is !!! abstract "Declarative ORM" UtilMeta has developed an efficient declarative ORM mechanism, also known as Scheme Query. We use `orm.Schema[User]` to define a Schema class with the User model injected, so that we can use the methods of the schema class to create, update, and serialize data. You can find more in [Data Query and ORM Document](../../guide/schema-query) -We can also find that a decorator named `@auth.session_config.plugin` is plug-in to the UserAPI class. This is the where the Session configuration is applied to the API. This plugin can save the Session data after each request and patch the response with corresponding `Set-Cookie` header +We can also find that a decorator named `@auth.session_config.plugin` is plugin to the UserAPI class. This is where the Session configuration is applied to the API. This plugin can save the Session data after each request and patch the response with corresponding `Set-Cookie` header ### Login & Logout API @@ -243,7 +243,7 @@ class UserAPI(api.API): session.flush() ``` -In The login API, we call the `login()` method in our authentication configuration to complete the login simply. Since we have configured the login field and password field, the UtilMeta can help us complete the password verification and login automatically. If the login is successful, the corresponding user instance is returned. So we can throw an error if the `login()` result is None, and after a successful login, we can call `UserSchema.init` to return the login user data to the client. +In the login API, we call the `login()` method in our authentication configuration to complete the login simply. Since we have configured the login field and password field, the UtilMeta can help us complete the password verification and login automatically. If the login is successful, the corresponding user instance is returned. So we can throw an error if the `login()` result is None, and after a successful login, we can call `UserSchema.init` to return the login user data to the client. !!! tip The use of `login()` method is not mandatory, you can write your custom login logc if you need diff --git a/utilmeta/core/api/base.py b/utilmeta/core/api/base.py index 606570d..fbcf223 100644 --- a/utilmeta/core/api/base.py +++ b/utilmeta/core/api/base.py @@ -213,7 +213,7 @@ def _generate_routes(cls): # 2. @api.parser (method=None) # 3. def get(self): (method='get') # 4. @api(method='CUSTOM') (method='custom') - val = cls._endpoint_cls.apply_for(val) + val = cls._endpoint_cls.apply_for(val, cls) elif hook_type: val = cls._hook_cls.dispatch_for(val, hook_type) else: diff --git a/utilmeta/core/api/endpoint.py b/utilmeta/core/api/endpoint.py index 8cf4927..dd27311 100644 --- a/utilmeta/core/api/endpoint.py +++ b/utilmeta/core/api/endpoint.py @@ -1,6 +1,6 @@ from utilmeta import utils from utilmeta.utils import exceptions as exc -from typing import Callable, Union, TYPE_CHECKING +from typing import Callable, Union, Type, TYPE_CHECKING from utilmeta.utils.plugin import PluginTarget, PluginEvent from utilmeta.utils.error import Error from utilmeta.utils.context import ContextWrapper, Property @@ -52,7 +52,7 @@ def init_prop(self, prop: Property, val: ParserField): # noqa, to be inherit class Endpoint(PluginTarget): @classmethod - def apply_for(cls, func): + def apply_for(cls, func: Callable, api: Type['API'] = None): _cls = getattr(func, 'cls', None) if not _cls or not issubclass(_cls, Endpoint): # override current class @@ -65,12 +65,15 @@ def apply_for(cls, func): continue # func properties override the default kwargs kwargs[key] = v + if api: + kwargs.update(api=api) return _cls(func, **kwargs) parser_cls = FunctionParser wrapper_cls = RequestContextWrapper def __init__(self, f: Callable, *, + api: Type['API'] = None, method: str, plugins: list = None, idempotent: bool = None, @@ -83,6 +86,7 @@ def __init__(self, f: Callable, *, raise TypeError(f'Invalid endpoint function: {f}') self.f = f + self.api = api self.method = method self.idempotent = idempotent self.eager = eager @@ -101,6 +105,14 @@ def iter_plugins(self): def getattr(self, name: str, default=None): return getattr(self.f, name, default) + @property + def ref(self) -> str: + if self.api: + return f'{self.api.__ref__}.{self.f.__name__}' + if self.module_name: + return f'{self.module_name}.{self.f.__name__}' + return self.f.__name__ + @property def module_name(self): return getattr(self.f, '__module__', None) diff --git a/utilmeta/core/api/specs/openapi.py b/utilmeta/core/api/specs/openapi.py index b15679d..0dbb12c 100644 --- a/utilmeta/core/api/specs/openapi.py +++ b/utilmeta/core/api/specs/openapi.py @@ -400,10 +400,13 @@ def parse_properties(self, props: Dict[str, ParserProperty]) -> Tuple[dict, dict 'in': _in, 'name': key, 'required': prop.required, - 'description': prop.description, - 'deprecated': prop.deprecated, 'schema': generator(), } + if prop.description: + data['description'] = prop.description + if prop.deprecated: + data['deprecated'] = True + if isinstance(field.field, properties.RequestParam): if field.field.style: data.update(style=field.field.style) @@ -512,6 +515,8 @@ def from_endpoint(self, endpoint: Endpoint, operation.update(parameters=list(params.values())) if body and endpoint.method in HAS_BODY_METHODS: operation.update(requestBody=body) + if endpoint.ref: + operation.update({'x-ref': endpoint.ref}) return operation def from_route(self, route: APIRoute, diff --git a/utilmeta/ops/client.py b/utilmeta/ops/client.py index 309e691..5d107f0 100644 --- a/utilmeta/ops/client.py +++ b/utilmeta/ops/client.py @@ -1,10 +1,12 @@ +import utype + from utilmeta.core.cli import Client from utilmeta.core import response, api from utilmeta.core import request from utype.types import * from .key import encrypt_data from .schema import NodeMetadata, SupervisorBasic, ServiceInfoSchema, SupervisorInfoSchema, \ - SupervisorData, ResourcesSchema + SupervisorData, ResourcesSchema, ResourcesData class SupervisorResponse(response.Response): @@ -43,6 +45,21 @@ def success(self): return False +class SupervisorResourcesResponse(SupervisorResponse): + name = 'resources' + result: ResourcesData + + +class NodeData(utype.Schema): + node_id: str + url: str + + +class SupervisorNodeResponse(SupervisorResponse): + name = 'add_node' + result: NodeData + + # class AddNodeResponse(SupervisorResponse): # name = 'info' # result: InfoSchema @@ -50,10 +67,10 @@ def success(self): class SupervisorClient(Client): @api.post('/') - def add_node(self, data: NodeMetadata = request.Body) -> SupervisorResponse: pass + def add_node(self, data: NodeMetadata = request.Body) -> SupervisorNodeResponse: pass @api.post('/resources') - def upload_resources(self, data: ResourcesSchema = request.Body) -> SupervisorResponse: pass + def upload_resources(self, data: ResourcesSchema = request.Body) -> SupervisorResourcesResponse: pass @api.get('/list') def get_supervisors(self) -> SupervisorListResponse: pass @@ -115,4 +132,4 @@ def process_request(self, req: request.Request): class OperationsClient(Client): @api.post('/') - def add_supervisor(self, data: SupervisorData = request.Body) -> ServiceInfoSchema: pass + def add_supervisor(self, data: SupervisorData = request.Body) -> ServiceInfoResponse: pass diff --git a/utilmeta/ops/connect.py b/utilmeta/ops/connect.py index 9e4309b..08c0230 100644 --- a/utilmeta/ops/connect.py +++ b/utilmeta/ops/connect.py @@ -50,6 +50,7 @@ def fetch_supervisor_info(base_url: str): ts_pairs.sort(key=lambda v: v[1]) return ts_pairs[0][0] + def connect_supervisor( key: str, base_url: str = None, @@ -87,6 +88,7 @@ def connect_supervisor( ops_api=ops_api ) resources = ResourcesManager(service) + url = None try: with SupervisorClient( @@ -103,10 +105,17 @@ def connect_supervisor( # update after if not supervisor_obj.node_id or not supervisor_obj.public_key: raise ValueError('supervisor failed to create') - + if supervisor_obj.node_id != resp.result.node_id: + raise ValueError(f'supervisor failed to create: inconsistent node id: ' + f'{supervisor_obj.node_id}, {resp.result.node_id}') + url = resp.result.url print(f'supervisor[{supervisor_obj.node_id}] created') except Exception as e: supervisor_obj.delete() raise e resources.sync_resources(supervisor_obj) + + print('supervisor connected successfully!') + if url: + print(f'please visit {url} to view and manage your APIs') diff --git a/utilmeta/ops/resources.py b/utilmeta/ops/resources.py index e429cfa..26a8bd0 100644 --- a/utilmeta/ops/resources.py +++ b/utilmeta/ops/resources.py @@ -3,8 +3,9 @@ from .models import Supervisor, Resource from .config import Operations from .schema import NodeMetadata, ResourcesSchema, \ - InstanceSchema, ServerSchema, TableSchema, DatabaseSchema, CacheSchema + InstanceSchema, ServerSchema, TableSchema, DatabaseSchema, CacheSchema, ResourceData from utilmeta import UtilMeta +from utilmeta.utils import fast_digest, json_dumps class ModelGenerator: @@ -268,7 +269,7 @@ def get_caches(self): def get_tasks(self): pass - def get_resources(self, node_id, etag: str = None) -> ResourcesSchema: + def get_resources(self, node_id, etag: str = None) -> Optional[ResourcesSchema]: from utilmeta.core.api.specs.openapi import OpenAPI from utilmeta import service openapi = OpenAPI(service)() @@ -290,9 +291,64 @@ def get_resources(self, node_id, etag: str = None) -> ResourcesSchema: caches=self.get_caches() ) if etag: - pass + resources_etag = fast_digest( + json_dumps(data), + compress=True, + case_insensitive=False + ) + if etag == resources_etag: + return None return data + def save_resources(self, resources: List[ResourceData], supervisor: Supervisor): + remote_pk_map = {val['remote_id']: val['pk'] for val in Resource.objects.filter( + node_id=supervisor.node_id, + ).values('pk', 'remote_id')} + + remote_pks = [] + updates = [] + creates = [] + for resource in resources: + remote_pks.append(resource.remote_id) + if resource.remote_id in remote_pk_map: + updates.append( + Resource( + id=remote_pk_map[resource.remote_id], + service=self.service.name, + node_id=supervisor.node_id, + deleted=False, + **resource + ) + ) + else: + creates.append( + Resource( + service=self.service.name, + node_id=supervisor.node_id, + **resource + ) + ) + + if updates: + Resource.objects.bulk_update( + updates, + fields=['server_id', 'ident', 'route', 'deleted'], + ) + if creates: + Resource.objects.bulk_create( + creates, + fields=['server_id', 'ident', 'route', 'deleted'], + ignore_conflicts=True + ) + + Resource.objects.filter( + node_id=supervisor.node_id, + ).exclude( + remote_id__in=remote_pks + ).update( + deleted=True + ) + def sync_resources(self, supervisor: Supervisor = None, force: bool = False): from utilmeta import service ops_config = Operations.config() @@ -322,6 +378,19 @@ def sync_resources(self, supervisor: Supervisor = None, force: bool = False): resp = client.upload_resources( data=resources ) + if resp.status == 304: + continue if not resp.success: raise ValueError(f'sync to supervisor[{supervisor.node_id}]' f' failed with error: {resp.message}') + + if resp.result.resources_etag: + supervisor.resources_etag = resp.result.resources_etag + supervisor.save(update_fields=['resources_etag']) + + self.save_resources( + resp.result.resources, + supervisor=supervisor + ) + + print(f'sync resources to supervisor[{supervisor.node_id}] successfully') diff --git a/utilmeta/ops/schema.py b/utilmeta/ops/schema.py index e28158d..b7e31bd 100644 --- a/utilmeta/ops/schema.py +++ b/utilmeta/ops/schema.py @@ -3,6 +3,7 @@ from utype.types import * from . import __spec_version__ import utilmeta +from utilmeta.core.api.specs.openapi import OpenAPISchema class SupervisorBasic(Schema): @@ -70,7 +71,7 @@ class TableSchema(ResourceBase): class ServerSchema(ResourceBase): ip: str # public_ip: Optional[str] = None - domain: Optional[str] = None + # domain: Optional[str] = None system: str platform: dict = Field(default_factory=dict) @@ -80,6 +81,8 @@ class ServerSchema(ResourceBase): cpu_num: int memory_total: int disk_total: int + max_open_files: Optional[int] = None + max_socket_conn: Optional[int] = None devices: dict = Field(default_factory=dict) @@ -121,7 +124,7 @@ class CacheSchema(ResourceBase): class ResourcesSchema(Schema): metadata: NodeMetadata - openapi: dict = Field(default_factory=None) + openapi: Optional[OpenAPISchema] = Field(default_factory=None) tables: List[TableSchema] = Field(default_factory=list) # model @@ -129,3 +132,16 @@ class ResourcesSchema(Schema): databases: List[DatabaseSchema] = Field(default_factory=list) caches: List[CacheSchema] = Field(default_factory=list) tasks: list = Field(default_factory=list) + + +class ResourceData(utype.Schema): + remote_id: str + server_id: Optional[str] = utype.Field(default=None, defer_default=True) + type: str + ident: str + route: str + + +class ResourcesData(utype.Schema): + resources: List[ResourceData] + resources_etag: str