add resources saving in ops

README.md

@@ -36,7 +36,7 @@ Using the declarative power from UtilMeta, you can easily write APIs with auto r
 UtilMeta developed a standard that support all major Python web framework like **django**, **flask**, **fastapi** (starlette), **sanic**, **tornado** as runtime backend, and support current projects using these frameworks to develop new API using UtilMeta progressively
 <img src="https://utilmeta.com/img/py.section2.png" href="https://utilmeta.com/py" target="_blank"  alt="drawing" width="720"/>
 ### Highly Flexible & Extensible
-UtilMeta is highly flexible with a series of plugins includes authentication (Session/JWT), CORS, rate limit, retry, and can be extended to support more features.
+UtilMeta is highly flexible with a series of plugins including authentication (Session/JWT), CORS, rate limit, retry, and can be extended to support more features.
 
 ### Full-lifecycle DevOps Solution
 The [UtilMeta Platform](https://utilmeta.com/) provided the full-lifecycle DevOps solution for this framework, the API Docs, Debug, Logs, Monitoring, Alerts, Analytics will all been taken care of in the platform
@@ -126,7 +126,7 @@ if you request the ArticleAPI like `GET /article?id=1`, you will get the result
   "content": "hello world"
 }
 ```
-This is conform to what you declared, and the OpenAPI docs will be generated automatically
+This conforms to what you declared, and the OpenAPI docs will be generated automatically
 ### Migrate
 
 Integrate current django/flask/fastapi/... project with UtilMeta API is as easy as follows 

docs/en/tutorials/user-auth.md

@@ -16,7 +16,7 @@ meta setup demo-user
 ```
 Enter `django` when prompted to select backend
 
-After the project is created, we need to configure the database connection of the service, open it `server.py`, and insert the following code
+After the project is created, we need to configure the database connection of the service, open `server.py`, and insert the following code
 
 ```python
 service = UtilMeta(...)
@@ -54,7 +54,7 @@ You can see that a new folder named `user` has been created in our project folde
 
 The `migrations` folder is where Django handles the database migrations of the models
 
-Once the app is created, we insert a line into the  Django settings of  `server.py` to specify the app.
+Once the app is created, we insert a line into the  `DjangoSettings` of  `server.py` to specify the app.
 
 ```python hl_lines="3"
 service.use(DjangoSettings(
@@ -67,7 +67,7 @@ So far, we have completed the configuration and initialization of the project.
 
 ## 2. Write user model
 
-The user’s login registration API, of course, revolves around the “user”. Before developing the API, we first write the user’s data model. We open `user/models.py` and write
+The user APIs depends on the "**user**", so before developing the API, we should write the user’s data model. We open `user/models.py` and write
 ```python
 from django.db import models
 from utilmeta.core.orm.backends.django.models import AbstractSession, PasswordField
@@ -146,11 +146,11 @@ user_config = auth.User(
 In this code, `SessionSchema` is the core engine that processes and stores Session data, `session_config`  declares the Session configuration with Session model and engine we just wrote, and configures the corresponding Cookie policy
 
 !!! tip
-	We use session store based on database to simply our tutorial, in practive, we often use cache+db as the store, you can find more in [Session Authentication](../../guide/auth#session)
+	We use session store based on database to simply our tutorial, in practice, we often use cache+db as the store, you can find more in [Session Authentication](../../guide/auth#session)
 
 We also declare the user authentication configuration `user_config` with the following params
 
-* `user_model` Specify the user model for authentication, which is the User model I wrote in the previous section.
+* `user_model`: Specify the user model for authentication, which is the User model I wrote in the previous section.
 * `authentication`: Specify the authentication method. We pass `session_config` in to declare that user authentication is performed using Session.
 * `key`: Specify the key of the current user ID in the session data
 * `login_fields`: Fields that can be used for login, such as username, email, etc., which need to be unique.
@@ -203,7 +203,7 @@ The logic in the signup API function is
 !!! abstract "Declarative ORM"
 	UtilMeta has developed an efficient declarative ORM mechanism, also known as Scheme Query. We use `orm.Schema[User]` to define a Schema class with the User model injected, so that we can use the methods of the schema class to create, update, and serialize data. You can find more in [Data Query and ORM Document](../../guide/schema-query)
 
-We can also find that a decorator named `@auth.session_config.plugin` is plug-in to the UserAPI class. This is the where the Session configuration is applied to the API. This plugin can save the Session data after each request and patch the response with corresponding `Set-Cookie` header
+We can also find that a decorator named `@auth.session_config.plugin` is plugin to the UserAPI class. This is where the Session configuration is applied to the API. This plugin can save the Session data after each request and patch the response with corresponding `Set-Cookie` header
 
 ### Login & Logout API
 
@@ -243,7 +243,7 @@ class UserAPI(api.API):
         session.flush()
 ```
 
-In The login API, we call the `login()` method in our authentication configuration to complete the login simply. Since we have configured the login field and password field, the UtilMeta can help us complete the password verification and login automatically. If the login is successful, the corresponding user instance is returned. So we can throw an error if the `login()` result is None, and after a successful login, we can call `UserSchema.init` to return the login user data to the client.
+In the login API, we call the `login()` method in our authentication configuration to complete the login simply. Since we have configured the login field and password field, the UtilMeta can help us complete the password verification and login automatically. If the login is successful, the corresponding user instance is returned. So we can throw an error if the `login()` result is None, and after a successful login, we can call `UserSchema.init` to return the login user data to the client.
 
 !!! tip
 	The use of `login()` method is not mandatory, you can write your custom login logc if you need

utilmeta/core/api/base.py

@@ -213,7 +213,7 @@ def _generate_routes(cls):
                     # 2. @api.parser             (method=None)
                     # 3. def get(self):          (method='get')
                     # 4. @api(method='CUSTOM')   (method='custom')
-                    val = cls._endpoint_cls.apply_for(val)
+                    val = cls._endpoint_cls.apply_for(val, cls)
                 elif hook_type:
                     val = cls._hook_cls.dispatch_for(val, hook_type)
                 else:

utilmeta/core/api/endpoint.py

@@ -1,6 +1,6 @@
 from utilmeta import utils
 from utilmeta.utils import exceptions as exc
-from typing import Callable, Union, TYPE_CHECKING
+from typing import Callable, Union, Type, TYPE_CHECKING
 from utilmeta.utils.plugin import PluginTarget, PluginEvent
 from utilmeta.utils.error import Error
 from utilmeta.utils.context import ContextWrapper, Property
@@ -52,7 +52,7 @@ def init_prop(self, prop: Property, val: ParserField):    # noqa, to be inherit
 
 class Endpoint(PluginTarget):
     @classmethod
-    def apply_for(cls, func):
+    def apply_for(cls, func: Callable, api: Type['API'] = None):
         _cls = getattr(func, 'cls', None)
         if not _cls or not issubclass(_cls, Endpoint):
             # override current class
@@ -65,12 +65,15 @@ def apply_for(cls, func):
                 continue
             # func properties override the default kwargs
             kwargs[key] = v
+        if api:
+            kwargs.update(api=api)
         return _cls(func, **kwargs)
 
     parser_cls = FunctionParser
     wrapper_cls = RequestContextWrapper
 
     def __init__(self, f: Callable, *,
+                 api: Type['API'] = None,
                  method: str,
                  plugins: list = None,
                  idempotent: bool = None,
@@ -83,6 +86,7 @@ def __init__(self, f: Callable, *,
             raise TypeError(f'Invalid endpoint function: {f}')
 
         self.f = f
+        self.api = api
         self.method = method
         self.idempotent = idempotent
         self.eager = eager
@@ -101,6 +105,14 @@ def iter_plugins(self):
     def getattr(self, name: str, default=None):
         return getattr(self.f, name, default)
 
+    @property
+    def ref(self) -> str:
+        if self.api:
+            return f'{self.api.__ref__}.{self.f.__name__}'
+        if self.module_name:
+            return f'{self.module_name}.{self.f.__name__}'
+        return self.f.__name__
+
     @property
     def module_name(self):
         return getattr(self.f, '__module__', None)

utilmeta/core/api/specs/openapi.py

@@ -400,10 +400,13 @@ def parse_properties(self, props: Dict[str, ParserProperty]) -> Tuple[dict, dict
                         'in': _in,
                         'name': key,
                         'required': prop.required,
-                        'description': prop.description,
-                        'deprecated': prop.deprecated,
                         'schema': generator(),
                     }
+                    if prop.description:
+                        data['description'] = prop.description
+                    if prop.deprecated:
+                        data['deprecated'] = True
+
                     if isinstance(field.field, properties.RequestParam):
                         if field.field.style:
                             data.update(style=field.field.style)
@@ -512,6 +515,8 @@ def from_endpoint(self, endpoint: Endpoint,
             operation.update(parameters=list(params.values()))
         if body and endpoint.method in HAS_BODY_METHODS:
             operation.update(requestBody=body)
+        if endpoint.ref:
+            operation.update({'x-ref': endpoint.ref})
         return operation
 
     def from_route(self, route: APIRoute,

utilmeta/ops/client.py

@@ -1,10 +1,12 @@
+import utype
+
 from utilmeta.core.cli import Client
 from utilmeta.core import response, api
 from utilmeta.core import request
 from utype.types import *
 from .key import encrypt_data
 from .schema import NodeMetadata, SupervisorBasic, ServiceInfoSchema, SupervisorInfoSchema, \
-    SupervisorData, ResourcesSchema
+    SupervisorData, ResourcesSchema, ResourcesData
 
 
 class SupervisorResponse(response.Response):
@@ -43,17 +45,32 @@ def success(self):
         return False
 
 
+class SupervisorResourcesResponse(SupervisorResponse):
+    name = 'resources'
+    result: ResourcesData
+
+
+class NodeData(utype.Schema):
+    node_id: str
+    url: str
+
+
+class SupervisorNodeResponse(SupervisorResponse):
+    name = 'add_node'
+    result: NodeData
+
+
 # class AddNodeResponse(SupervisorResponse):
 #     name = 'info'
 #     result: InfoSchema
 
 
 class SupervisorClient(Client):
     @api.post('/')
-    def add_node(self, data: NodeMetadata = request.Body) -> SupervisorResponse: pass
+    def add_node(self, data: NodeMetadata = request.Body) -> SupervisorNodeResponse: pass
 
     @api.post('/resources')
-    def upload_resources(self, data: ResourcesSchema = request.Body) -> SupervisorResponse: pass
+    def upload_resources(self, data: ResourcesSchema = request.Body) -> SupervisorResourcesResponse: pass
 
     @api.get('/list')
     def get_supervisors(self) -> SupervisorListResponse: pass
@@ -115,4 +132,4 @@ def process_request(self, req: request.Request):
 
 class OperationsClient(Client):
     @api.post('/')
-    def add_supervisor(self, data: SupervisorData = request.Body) -> ServiceInfoSchema: pass
+    def add_supervisor(self, data: SupervisorData = request.Body) -> ServiceInfoResponse: pass

utilmeta/ops/connect.py

@@ -50,6 +50,7 @@ def fetch_supervisor_info(base_url: str):
     ts_pairs.sort(key=lambda v: v[1])
     return ts_pairs[0][0]
 
+
 def connect_supervisor(
     key: str,
     base_url: str = None,
@@ -87,6 +88,7 @@ def connect_supervisor(
         ops_api=ops_api
     )
     resources = ResourcesManager(service)
+    url = None
 
     try:
         with SupervisorClient(
@@ -103,10 +105,17 @@ def connect_supervisor(
             # update after
             if not supervisor_obj.node_id or not supervisor_obj.public_key:
                 raise ValueError('supervisor failed to create')
-
+            if supervisor_obj.node_id != resp.result.node_id:
+                raise ValueError(f'supervisor failed to create: inconsistent node id: '
+                                 f'{supervisor_obj.node_id}, {resp.result.node_id}')
+            url = resp.result.url
             print(f'supervisor[{supervisor_obj.node_id}] created')
     except Exception as e:
         supervisor_obj.delete()
         raise e
 
     resources.sync_resources(supervisor_obj)
+
+    print('supervisor connected successfully!')
+    if url:
+        print(f'please visit {url} to view and manage your APIs')

utilmeta/ops/resources.py

@@ -3,8 +3,9 @@
 from .models import Supervisor, Resource
 from .config import Operations
 from .schema import NodeMetadata, ResourcesSchema, \
-    InstanceSchema, ServerSchema, TableSchema, DatabaseSchema, CacheSchema
+    InstanceSchema, ServerSchema, TableSchema, DatabaseSchema, CacheSchema, ResourceData
 from utilmeta import UtilMeta
+from utilmeta.utils import fast_digest, json_dumps
 
 
 class ModelGenerator:
@@ -268,7 +269,7 @@ def get_caches(self):
     def get_tasks(self):
         pass
 
-    def get_resources(self, node_id, etag: str = None) -> ResourcesSchema:
+    def get_resources(self, node_id, etag: str = None) -> Optional[ResourcesSchema]:
         from utilmeta.core.api.specs.openapi import OpenAPI
         from utilmeta import service
         openapi = OpenAPI(service)()
@@ -290,9 +291,64 @@ def get_resources(self, node_id, etag: str = None) -> ResourcesSchema:
             caches=self.get_caches()
         )
         if etag:
-            pass
+            resources_etag = fast_digest(
+                json_dumps(data),
+                compress=True,
+                case_insensitive=False
+            )
+            if etag == resources_etag:
+                return None
         return data
 
+    def save_resources(self, resources: List[ResourceData], supervisor: Supervisor):
+        remote_pk_map = {val['remote_id']: val['pk'] for val in Resource.objects.filter(
+            node_id=supervisor.node_id,
+        ).values('pk', 'remote_id')}
+
+        remote_pks = []
+        updates = []
+        creates = []
+        for resource in resources:
+            remote_pks.append(resource.remote_id)
+            if resource.remote_id in remote_pk_map:
+                updates.append(
+                    Resource(
+                        id=remote_pk_map[resource.remote_id],
+                        service=self.service.name,
+                        node_id=supervisor.node_id,
+                        deleted=False,
+                        **resource
+                    )
+                )
+            else:
+                creates.append(
+                    Resource(
+                        service=self.service.name,
+                        node_id=supervisor.node_id,
+                        **resource
+                    )
+                )
+
+        if updates:
+            Resource.objects.bulk_update(
+                updates,
+                fields=['server_id', 'ident', 'route', 'deleted'],
+            )
+        if creates:
+            Resource.objects.bulk_create(
+                creates,
+                fields=['server_id', 'ident', 'route', 'deleted'],
+                ignore_conflicts=True
+            )
+
+        Resource.objects.filter(
+            node_id=supervisor.node_id,
+        ).exclude(
+            remote_id__in=remote_pks
+        ).update(
+            deleted=True
+        )
+
     def sync_resources(self, supervisor: Supervisor = None, force: bool = False):
         from utilmeta import service
         ops_config = Operations.config()
@@ -322,6 +378,19 @@ def sync_resources(self, supervisor: Supervisor = None, force: bool = False):
                 resp = client.upload_resources(
                     data=resources
                 )
+                if resp.status == 304:
+                    continue
                 if not resp.success:
                     raise ValueError(f'sync to supervisor[{supervisor.node_id}]'
                                      f' failed with error: {resp.message}')
+
+                if resp.result.resources_etag:
+                    supervisor.resources_etag = resp.result.resources_etag
+                    supervisor.save(update_fields=['resources_etag'])
+
+                self.save_resources(
+                    resp.result.resources,
+                    supervisor=supervisor
+                )
+
+                print(f'sync resources to supervisor[{supervisor.node_id}] successfully')