[Big change] Server refactor (Part 1) #858

da-h · 2022-07-29T20:30:42Z

Description

There has already been done lots of work regarding the refactoring and restructuring of the server code (see #675).
The goal of this PR is to merge these changes with the current server version to make room for further improvements on top of that. And most importantly: not loosing work in the long-run that has already been done.

(Note: Even though I tagged this PR as "Part 1", the changes suggested are still fully functional).

To accomplish the merge of #675

First, I basically cherry-picked the commits containing the actual work (i.e. excluding the merge-commits). This has the side-effect that the original contribution is maintained in the logs.
Then I've added revert-commits on top of that to essentially create a non-OP on-top of master.
After that, I reapplied the refactoring applied in [Big change][WIP] Server refactor #675 and saved the changes as a new commit named reapply all changes since server refactor.
Squashing that commit with the revert-commit basically results in a rebase of master onto [Big change][WIP] Server refactor #675.
(The other way round would be possible as well, but I felt this was easier in this case).

Some minor changes on the way:

changed the method get_visdom_path to use importlib instead of inspect to get the actual file name of static files. (I found the previous version could have caching-problems with static files during development if I remember correctly).
Moved LAYOUT_FILE and MAX_SOCKET_WAIT to defaults.py (to match its location to that of the other global uppercased config variables).
Changed wildcard imports (from X import * to specific imports).

Note:
I needed to add a change in the github workflow just for this PR: As the file structure changed (the executable python file is py/visdom/server instead of py/visdom/server.py), the startup of the server would not work anymore for the "init"-stage of the cypress tests.
This (last) commit can basically be reverted after or removed before the merge.

How Has This Been Tested?

Just did a short test myself using the demo + tested against default cypress integration tests.

Types of changes

Refactor
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

I adapted the version number under py/visdom/VERSION according to Semantic Versioning
Changed from 0.2.1 to 0.2.2.
My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.

da-h · 2022-07-29T20:38:07Z

Regarding the Socket-Wrappers & Handlers:
I have added a short explanation regarding the naming scheme of the socket wrappers (VisSocketWrapper, SocketHandler, ...) as code comments.
If I understood correctly, the VisSocket-Handlers are for clients that mainly write data, while the Socket-Handlers are for clients that mainly read data (for instance for visualization).

One question from my side: Why has this seperation been done during development?
Is it possibly an authentication reason or maybe a performance reason? I did not found any priorization or more performant connections for one or the other, but I didn't look too closely for such a feature.

I ask because I thought about merging these two types of handlers in case they are not actually handled differently.
I think this could simplify the code quite a bit.

JackUrb · 2022-07-29T20:47:48Z

Excited to see these changes revived!

You're right on the money for why there were different types of socket handlers. Back when we were driving internal visdom adoption we needed separate flows for writing to the server and reading from it for authentication reasons. The keepalive setups for each were different, and so we had to make a drop-in replacement for just one of the two paths. Nowadays we don't use this setup the same way anymore, so I'm happy to have things consolidated.

da-h · 2022-07-29T21:05:53Z

Ah I see, thanks for the additional information! :)

My tendency is not to simplify existing structures if they have a good reason like here. On the other hand, the code is indeed more complicated due to the split. Additionally, if nowadays more people need such a feature, probably a more complex role distribution (in contrast to just separating read and write) could become a long-term goal.

da-h · 2022-09-20T05:06:12Z

Rebased onto master to retrigger actions with the changes of #866 in place.

da-h · 2022-10-06T22:49:59Z

Not sure yet, how to handle version-numbers correctly, especially for multiple parallel open PRs, or the question if every small PR should actually increase the version.

Nevertheless, larger code changes such as this one could be given a new version number, i guess.
Thus, I have increased the version by a 0.0.1 step.

da-h · 2022-10-06T22:59:00Z

I am sorry, did a mistake here in the "force-push" history by rebasing onto the wrong branch.

Last push sits correctly on top of master / v0.2.1.

da-h · 2022-10-17T19:13:23Z

(Just a rebase onto current master).

da-h · 2022-10-17T20:42:52Z

Was missing #873 PR.

Latest push contains that one as well. "Compare to previous Push"-Button should now show exactly those changes now.
(Did not see the conflict due to the whole file-restructuring of this rather large PR).

I am thinking what I could do to provide a clearer way to show if this PR is actually non-destructive somewhere. Possibly a script that reverts the split into sub-files?
Merging the files back programmatically to result in a file very similar to the current master's py/visdom/server.py should be possible.

JackUrb · 2022-10-17T20:58:40Z

If this is still functional as expected (which, given the tests, it likely is) I'm happy to accept this sooner rather than later (instead of building out a solution to constantly keep it updated).

da-h · 2022-10-17T21:05:58Z

Sure!
Just to mention it: the easier-to-review alternative is to split this up into two:

Separating the changes that you have done in [Big change][WIP] Server refactor #675 plus the one commit afterwards reapply all changes since server refactor. This PR is the main work, structure-wise where it's much easier to have a script that checks differences to master by merging the separated files again.
A second PR could contain the refactoring-approaches afterwards.

For both ways, merging this big PR, or, separating the two parts mentioned, I am happy to look into bugs that might have been introduced by me here.

JackUrb · 2022-10-17T21:12:03Z

I'll take a review pass tomorrow - given #675 was mostly me directly copy-pasting into smaller files and then making changes on top, I'm fairly convinced that the majority of the underlying server code is present as it was, so I'll do a review on the following content.

da-h · 2022-10-17T21:21:23Z

I didn't mean to push you. :)
A change of this size obviously needs some time to review.

JackUrb · 2022-10-17T21:24:34Z

No worries, I've left this one sitting open for far too long 😅

da-h · 2022-10-17T21:43:32Z

See a minor mistake that I will fix tomorrow morning: all the file headers say "Facebook Inc." instead of "Visdom Team". 😅

JackUrb

All-in-all this is a great set of changes. Thanks for the additional work to extend my original set to still be valid against the current codebase. I feel like this separation does a lot to make the inner workings of Visdom more approachable.

JackUrb · 2022-10-18T23:25:03Z

py/visdom/extra_deps/__init__.py

@@ -1,5 +0,0 @@
-# Copyright 2017-present, The Visdom Authors


Why do we drop this file? The core function of it is to retain the directory, wherein people can import optional dependencies for Visdom. Realistically, this is only useful for the bhtsne library, but that may still be in use. Perhaps instead of just an __init__.py, it makes sense to have a README.md too, but that's out of scope for this PR.

Ah, didn't know that. I've removed the deletion in the latest push below.

Hm I'm not seeing the most recent push. I'll merge this in once it's there!

Yes, I totally forgot to push, but it should be there now.

JackUrb · 2022-10-18T23:26:05Z

py/visdom/__init__.py

@@ -6,6 +6,8 @@
 # This source code is licensed under the license found in the


As an aside, it's not the best that the entirety of visdom lives in the __init__.py file too. For a future refactor 😉

Yeah, that's true; the python-client code grew out of control. ^^

Considering some of the issues, there are quite some bugs to fix there as well, but tons of tiny things to keep track of at the same time. Any Idea how to improve the code efficiently?

For things like this, I'd argue that visdom has reached a scale that we need to hit on more DevX and codebase quality. Before jumping into the small changes, reorganizing would come first. Then I'd probably want to start using typing, though that can happen in parallel with small changes (every time you take a pass through to fix a bug, doing code quality improvements in the same pass). Typing and testing make sure that as you're making changes, the code is to some degree keeping track of things by itself so you don't have to.

py/visdom/server/__main__.py

py/visdom/utils/shared_utils.py

setup.py

JackUrb · 2022-10-18T23:43:10Z

py/visdom/server/handlers/socket_handlers.py

+# ============== #
+# About & Naming #
+# ============== #
+
+# 1. *Handler- & *Wrap-classes are intended to have the same functionality
+#   - *Handler (e.g. VisSocketHandler) use WebSockets
+#   - *Wrap (e.g. VisSocketWrap) use polling-based connections instead
+#   - *Wrapper (e.g. VisSocketWrapper) is just a helper class for the respective *Wrap-class 
+#     to process the current state (instead of the state at the time of polling)
+# 2. VisSocket* classes (VisSocketHandler, VisSocketWrap & VisSocketWrapper)
+#   Their goal is to register clients with write access of actual data.
+# 3. Socket* classes (SocketHandler, SocketWrap & SocketWrapper)
+#   Their goal is to register clients with read access of data.
+#   Write access is limited to data and view organization (i.e. layout settings, env removal and env saving)


Thanks for providing the demystifying details for these :)

Yeah, wrapping my head around the code here was a bit tough. I felt that placing my notes here for others is surely helpful.

--

Maybe here is a good place to ask:
I am a bit unsure of how big of a PR one would like to have for changes like that, say, a refactoring of a large file. ;)

For instance, I do have worked a bit further on that file here; I merged the duplicated code as much as possible and placed the classes in a hierarchy, but felt that the change is altogether too much for one single PR.
Of course, larger code changes need more time to review, but on the other hand it does not make to much sense to split changes up too much, does it?

Definitely too much for one PR. I think that a good PR can be large, so long as it's easily summarized. My rule of thumb is the simple sentence summary:
"Splitting up the server into related files, but retaining functionality" makes sense.
"Refactoring socket code to reduce duplicated code" also makes sense.
Putting these two together may be too much.

…scheme

…aster work in PR

da-h mentioned this pull request Sep 17, 2022

Install visdom without Internet connection #865

Closed

da-h force-pushed the server_code_refactor branch from 3875f19 to b1825cc Compare September 20, 2022 05:05

da-h force-pushed the server_code_refactor branch 4 times, most recently from e6a1e10 to 2989507 Compare October 6, 2022 22:56

da-h mentioned this pull request Oct 14, 2022

cannot run "python -m visdom.server" #874

Open

da-h force-pushed the server_code_refactor branch from 2989507 to 4000d8f Compare October 17, 2022 19:12

da-h force-pushed the server_code_refactor branch from 4000d8f to f362176 Compare October 17, 2022 20:36

da-h force-pushed the server_code_refactor branch from f362176 to 015578d Compare October 18, 2022 06:09

JackUrb approved these changes Oct 18, 2022

View reviewed changes

JackUrb and others added 5 commits October 19, 2022 09:16

Refactoring server.py into more intentional files

80949fc

Splitting all handlers into socket and base handlers

802c612

reapply all changes since server refactor

fb3541f

server: simplify & fix get_visdom_path/_to

75c63e3

server: move LAYOUT_FILE to defaults.py

26a9ef1

David Hartmann added 10 commits October 19, 2022 09:16

server: cleanup of imports

8bdb9c1

setup: adapt console_script to new file structure

277bcbf

server: remove zmq (as it has not been used & zmq.ioloop is deprecated)

d3ce2d3

server: (minor) changed order of imports

b3d838a

server: move MAX_SOCKET_WAIT to defaults.py

f597e9e

server: added implementation comments about Sockets/Handlers

2bbe4c7

server: renamed ClientSocketWrapper -> SocketWrapper to unify naming …

77d5931

…scheme

github: adapted workflows to new filestructur in py/visdom/server

fd70fcd

github: (may be reverted after merge) ensures that cypress tests on m…

c573028

…aster work in PR

release version 0.2.2

f54852f

da-h force-pushed the server_code_refactor branch from 015578d to f54852f Compare October 19, 2022 17:14

JackUrb merged commit c26ca42 into fossasia:master Oct 19, 2022

da-h deleted the server_code_refactor branch October 19, 2022 19:00

This was referenced Oct 20, 2022

Add missing __init__.py files & let setup.py find all importable packages on its own #879

Merged

github-actions: adapt cypress test to new server code #880

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Big change] Server refactor (Part 1) #858

[Big change] Server refactor (Part 1) #858

da-h commented Jul 29, 2022 •

edited

da-h commented Jul 29, 2022 •

edited

JackUrb commented Jul 29, 2022

da-h commented Jul 29, 2022 •

edited

da-h commented Sep 20, 2022 •

edited

da-h commented Oct 6, 2022

da-h commented Oct 6, 2022

da-h commented Oct 17, 2022

da-h commented Oct 17, 2022 •

edited

JackUrb commented Oct 17, 2022

da-h commented Oct 17, 2022

JackUrb commented Oct 17, 2022

da-h commented Oct 17, 2022

JackUrb commented Oct 17, 2022

da-h commented Oct 17, 2022

JackUrb left a comment

JackUrb Oct 18, 2022

da-h Oct 19, 2022

JackUrb Oct 19, 2022

da-h Oct 19, 2022

JackUrb Oct 18, 2022

da-h Oct 19, 2022 •

edited

JackUrb Oct 19, 2022

JackUrb Oct 18, 2022

da-h Oct 19, 2022 •

edited

JackUrb Oct 19, 2022

		@@ -6,6 +6,8 @@
		# This source code is licensed under the license found in the

[Big change] Server refactor (Part 1) #858

[Big change] Server refactor (Part 1) #858

Conversation

da-h commented Jul 29, 2022 • edited

Description

How Has This Been Tested?

Types of changes

Checklist:

da-h commented Jul 29, 2022 • edited

JackUrb commented Jul 29, 2022

da-h commented Jul 29, 2022 • edited

da-h commented Sep 20, 2022 • edited

da-h commented Oct 6, 2022

da-h commented Oct 6, 2022

da-h commented Oct 17, 2022

da-h commented Oct 17, 2022 • edited

JackUrb commented Oct 17, 2022

da-h commented Oct 17, 2022

JackUrb commented Oct 17, 2022

da-h commented Oct 17, 2022

JackUrb commented Oct 17, 2022

da-h commented Oct 17, 2022

JackUrb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

da-h Oct 19, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

da-h Oct 19, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

da-h commented Jul 29, 2022 •

edited

da-h commented Jul 29, 2022 •

edited

da-h commented Jul 29, 2022 •

edited

da-h commented Sep 20, 2022 •

edited

da-h commented Oct 17, 2022 •

edited

da-h Oct 19, 2022 •

edited

da-h Oct 19, 2022 •

edited