API: support and profile isolve workflow

[mguidon]

1. Jobs that are only created, but not run do not show up in job listing (this might be useful for testing)
2. What is the sorting order for jobs?
3. Is there a rate limit? I have the impression starting jobs take quite some time. Could also be my implementation of the client. Do we have a number for the trhoughput?

Tasks

Beta Give feedback

1. bug: enums inputs break input matching #4793
   High Priority bug +2
   
2. Investigate bottleneck when creating many (more than 100) iSolve jobs at the same time
3. Setup profiling tool for the api server to detect performance bottlenecks
4. bug: job pagination and offset mismatch https://monitoring.osparc-master.speag.com/graylog/messages/graylog_6/d4db1092-5bc7-11ee-800a-0242ac130016
5. bug: delete error handling

[pcrespov]

@mguidon these are some answers

1. it should. list_jobs shows all created job. Perhaps the problem is that is limited to the first 20 jobs. Now the new pagination allows getting them all. We are also adding a filter functionality in API server: Allow filtering in paginated methods #4597
2. same order as projects: query.order_by(desc(projects.c.last_change_date), projects.c.id), i.e. last changed and order of creation. Perhaps we should add an option as well. Would you suggest a different option?
3. Yes, the traefik reverse proxy in front activates a rate-limiter for queries fot he api-server. see details of throughput in

   osparc-simcore/services/docker-compose.yml

   Lines 891 to 898 in 74636e0

   	# for each service in the stack a new middlaware for rate limiting needs to be registered here
   	# requests = average / period this is how the limits are defined
   	- traefik.http.middlewares.ratelimit-${SWARM_STACK_NAME}_api-server.ratelimit.average=1
   	- traefik.http.middlewares.ratelimit-${SWARM_STACK_NAME}_api-server.ratelimit.period=1m
   	# a burst is computed over a period of 1 second
   	- traefik.http.middlewares.ratelimit-${SWARM_STACK_NAME}_api-server.ratelimit.burst=10
   	# X-Forwarded-For header extracts second IP from the right, count starts at one
   	- traefik.http.middlewares.ratelimit-${SWARM_STACK_NAME}_api-server.ratelimit.sourcecriterion.ipstrategy.depth=2

4. Can you give me an example? In principle if it does not work in the GUI it might probably not work int he api.

[mguidon]

1. I will check again, maybe that has been changed/fixed
2. I have a test that checks the order. It randomly fails. But potentially the timing on server side might be different from my client that submits 10 jobs in a row one after the other.
3. I will have to look into what that means
4. Yes: in osparc.io, create a study with an ascent computational service and try to connect a file-picker. It might very well be that this is not very standard and just works for the frontend and we never had a problem because it is only used as a standalone service. Code is input_4 here

[elisabettai]

@pcrespov, I have also experienced the same issue that @mguidon reported regarding enums and input matching.

This is blocking us from creating services that have as inputs files AND enums. I've put more details in #4793. Let me know if I should deploy this service so that you can reproduce the error:

 File "/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/catalog/_api_units.py", line 73, in can_connect
    to_type = _get_type_name(to_input)
  File "/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/catalog/_api_units.py", line 17, in _get_type_name
    _type = port.content_schema["type"]
KeyError: 'type'

[bisgaard-itis]

Closing this after discussion with @mguidon as remaining tasks are not high priority