feat: python server #6

WinPlay02 · 2023-11-06T14:22:55Z

every execution needs a unique id to identify calculated values
executes compiled safe-ds code in memory (using a finder / loader)
pipelines are executed in child processes
communication from child processes to the main process uses a shared memory queue
allows connections via websocket (and http)
listens on a provided port (default 5000)
uses flask and flask-sock for HTTP and WS
uses gevent as WSGI implementation

feat: accept websocket & http connections test: added a partial test case for in memory execution Co-authored-by: Jonas B <97200640+smitedeluxe@users.noreply.github.com>

codecov · 2023-11-06T14:24:51Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (1cb1ba1) 100.00% compared to head (e949194) 100.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##              main        #6    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files            2         6     +4     
  Lines           13       317   +304     
==========================================
+ Hits            13       317   +304

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2023-11-06T14:25:19Z

🦙 MegaLinter status: ✅ SUCCESS

Descriptor	Linter	Files	Fixed	Errors	Elapsed time
✅ PYTHON	black	8	0	0	0.62s
✅ PYTHON	mypy	8		0	1.7s
✅ PYTHON	ruff	8	0	0	0.03s
✅ REPOSITORY	git_diff	yes		no	0.02s

See detailed report in MegaLinter reports
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by

…ain process using a queue

docs: added comments for functions

test: changed test to new execution

misc: disable linter for global variables + blind exception catching, where needed

test: mock-test all websocket interactions fix: use creation functions, where applicable

WinPlay02 · 2023-11-27T16:55:45Z

@lars-reimann Collecting coverage information across child processes (multiprocessing) does not seem to work.
I tried adding this section to .coveragerc:

[run]
parallel = True
concurrency = multiprocessing, thread

This also does not work, it only causes the testing process to freeze when calling multiprocessing.Manager() during setup.
Is there any (other) way to make coverage work with multiprocessing?

lars-reimann · 2023-11-27T18:06:11Z

I have no experience with this. Maybe the pytest-cov documentation helps.

…eems to be broken)

…o python-server # Conflicts: # tests/safeds_runner/server/test_websocket_mock.py

…ulation misc: remove main function and flask specific entry points from coverage calculation

…o python-server

WinPlay02 · 2023-11-28T01:23:13Z

It seems to be a problem only on Windows, as the CI passes. I annotated the tests skip execution on Windows for now.
Although they work when coverage calculation does not include multiprocessing (in .coveragerc, see above).

…tion resolution

tests/safeds_runner/server/test_pipeline_manager_type_conversion.py

src/safeds_runner/server/main.py

src/safeds_runner/server/__init__.py

src/safeds_runner/server/pipeline_manager.py

tests/safeds_runner/server/test_websocket_mock.py

refactor: use dataclasses as message storage instead of dicts test: make placeholder conversion test parametrized test: only skip multiprocessing tests on windows, when coverage is enabled

…o python-server # Conflicts: # src/safeds_runner/server/main.py # src/safeds_runner/server/pipeline_manager.py # tests/safeds_runner/server/test_websocket_mock.py

…server # Conflicts: # poetry.lock # pyproject.toml

Co-authored-by: Lars Reimann <mail@larsreimann.com>

…o python-server

WinPlay02 · 2023-11-30T00:20:28Z

I think I addressed all the suggestions. Now, the PipelineManager class takes the place of most of the global variables. It is created at startup and used in the websocket session.
But I couldn't get rid of the current_pipeline global in pipeline_manager.py as it is used for saving placeholders.

lars-reimann

Looks great overall. I just have some minor requests.

src/safeds_runner/server/pipeline_manager.py

refactor: hide attributes that don't need to be public

lars-reimann

Great contribution, thanks! 👍

## [0.2.0](v0.1.0...v0.2.0) (2023-11-30) ### Features * python server ([#6](#6)) ([a2c4f0f](a2c4f0f))

lars-reimann · 2023-11-30T14:12:24Z

🎉 This PR is included in version 0.2.0 🎉

The release is available on:

v0.2.0
GitHub release

Your semantic-release bot 📦🚀

feat: allow a program to be executed in memory

f13f6c4

feat: accept websocket & http connections test: added a partial test case for in memory execution Co-authored-by: Jonas B <97200640+smitedeluxe@users.noreply.github.com>

WinPlay02 requested a review from SmiteDeluxe November 6, 2023 14:22

WinPlay02 and others added 16 commits November 16, 2023 11:00

feat: handling exceptions when running pipeline

c453627

feat: execute pipelines in new subprocess and send messages back to m…

5eefe0f

…ain process using a queue

feat: rework placeholder saving

215df0b

refactor: put pipeline operations in their own module

9186587

docs: added comments for functions

misc: added some docstrings, types

5d6cda5

misc: added more docstrings, types

6887865

test: changed test to new execution

misc: added even more docstrings

2ee74aa

misc: added a few missing docstrings

6da854c

misc: disable linter for global variables + blind exception catching, where needed

misc: add missing type annotation to helper function

5d4d2b0

style: apply automated linter fixes

33aa2f6

style: apply automated linter fixes

05e1778

feat: expose "runner_save_placeholder" to client code

eecab94

test: mock-test all websocket interactions fix: use creation functions, where applicable

misc: linter fixes

214cc5b

style: linter fixes for test cases

8f58fe5

style: apply automated linter fixes

586ffe8

style: apply automated linter fixes

3649dda

WinPlay02 and others added 5 commits November 28, 2023 00:20

test: try multiprocessing coverage on ci and skip on windows (as it s…

95013f7

…eems to be broken)

Merge branch 'python-server' of https://github.com/Safe-DS/Runner int…

7e9716c

…o python-server # Conflicts: # tests/safeds_runner/server/test_websocket_mock.py

style: apply automated linter fixes

85e6534

misc: remove fallback paths that should not be hit from coverage calc…

0e56a5a

…ulation misc: remove main function and flask specific entry points from coverage calculation

Merge branch 'python-server' of https://github.com/Safe-DS/Runner int…

0f5e849

…o python-server

WinPlay02 and others added 2 commits November 28, 2023 02:33

feat: change "package" to "modulepath" and include it in module execu…

b82df25

…tion resolution

style: apply automated linter fixes

3dc7195

lars-reimann requested changes Nov 29, 2023

View reviewed changes

WinPlay02 and others added 16 commits November 29, 2023 16:30

refactor: extract message validation

125c7a0

refactor: use dataclasses as message storage instead of dicts test: make placeholder conversion test parametrized test: only skip multiprocessing tests on windows, when coverage is enabled

docs: Change comments to numpy format

d3538be

test: parametrize all tests

052a964

Merge branch 'python-server' of https://github.com/Safe-DS/Runner int…

3fee615

…o python-server # Conflicts: # src/safeds_runner/server/main.py # src/safeds_runner/server/pipeline_manager.py # tests/safeds_runner/server/test_websocket_mock.py

Merge branch 'main' of https://github.com/Safe-DS/Runner into python-…

ebd23be

…server # Conflicts: # poetry.lock # pyproject.toml

chore: recalculate poetry.lock hash

ad15629

docs: vscode extension -> VS Code extension

c058655

Co-authored-by: Lars Reimann <mail@larsreimann.com>

misc: fix linter warnings

4f28ab2

style: apply automated linter fixes

2cdb56c

style: apply automated linter fixes

0e66aec

misc: fix wrong coverage annotation

4dc542e

Merge branch 'python-server' of https://github.com/Safe-DS/Runner int…

727811b

…o python-server

feat: do not rely on globals as much, instead encapsulate in class

6380869

misc: fix linter warnings

0ce59dc

style: apply automated linter fixes

a9c47ce

style: apply automated linter fixes

6817517

WinPlay02 requested a review from lars-reimann November 30, 2023 00:20

lars-reimann requested changes Nov 30, 2023

View reviewed changes

src/safeds_runner/server/pipeline_manager.py Outdated Show resolved Hide resolved

src/safeds_runner/server/pipeline_manager.py Outdated Show resolved Hide resolved

src/safeds_runner/server/pipeline_manager.py Outdated Show resolved Hide resolved

WinPlay02 and others added 4 commits November 30, 2023 14:41

refactor: use pattern matching for python -> safeds type conversion

67d1ab5

refactor: hide attributes that don't need to be public

style: apply automated linter fixes

708a3e8

style: apply automated linter fixes

5b73a9a

fix: uncovered default path for type conversion pattern matching

e949194

lars-reimann approved these changes Nov 30, 2023

View reviewed changes

lars-reimann merged commit a2c4f0f into main Nov 30, 2023
8 checks passed

lars-reimann deleted the python-server branch November 30, 2023 13:58

lars-reimann pushed a commit that referenced this pull request Nov 30, 2023

chore(release): 0.2.0 [skip ci]

69aaded

## [0.2.0](v0.1.0...v0.2.0) (2023-11-30) ### Features * python server ([#6](#6)) ([a2c4f0f](a2c4f0f))

lars-reimann added the released Included in a release label Nov 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: python server #6

feat: python server #6

WinPlay02 commented Nov 6, 2023 •

edited

codecov bot commented Nov 6, 2023 •

edited

github-actions bot commented Nov 6, 2023 •

edited

WinPlay02 commented Nov 27, 2023

lars-reimann commented Nov 27, 2023

WinPlay02 commented Nov 28, 2023

WinPlay02 commented Nov 30, 2023

lars-reimann left a comment

lars-reimann left a comment •

edited

lars-reimann commented Nov 30, 2023

feat: python server #6

feat: python server #6

Conversation

WinPlay02 commented Nov 6, 2023 • edited

codecov bot commented Nov 6, 2023 • edited

Codecov Report

github-actions bot commented Nov 6, 2023 • edited

🦙 MegaLinter status: ✅ SUCCESS

WinPlay02 commented Nov 27, 2023

lars-reimann commented Nov 27, 2023

WinPlay02 commented Nov 28, 2023

WinPlay02 commented Nov 30, 2023

lars-reimann left a comment

Choose a reason for hiding this comment

lars-reimann left a comment • edited

Choose a reason for hiding this comment

lars-reimann commented Nov 30, 2023

WinPlay02 commented Nov 6, 2023 •

edited

codecov bot commented Nov 6, 2023 •

edited

github-actions bot commented Nov 6, 2023 •

edited

lars-reimann left a comment •

edited