SocketcandMedia Media class added for socketcand functionality #306

wiboticanders · 2023-07-11T23:50:44Z

Although this runs on PythonCAN, changing the PythonCANMedia class to function with socketcand is not very clean because socketcand has 2 additional variables compared to all the other PythonCAN compatible interfaces on pycyphal.
Socketcand can be referenced here: https://github.com/linux-can/socketcand , it is used for remote CAN access of udp socket. As far as copyright goes, because this is very similar to the PythonCANMedia class and I reused code from it I believe that the copyright still applies.

…cand interface

in init function

the old implimentation to a new one

added another error check for socketcand and CAN FD at same time

to file as well

_InterfaceParameters instead of separately in the FD and classic subclasses

This reverts commit d9f8267.

library functionality. It runs on Python-CAN like the interfaces in the PythonCANMedia class, but because it has extra variables (host and port), it is not very clean to just add socketcand to PythonCANMedia.

wiboticanders · 2023-07-11T23:56:13Z

Hello, can't tell why the read the docs build failed. Maybe because I added new files?

maksimdrachov · 2023-07-12T06:28:48Z

Regarding testing, it might be useful to take a look at the following forum post:

https://forum.opencyphal.org/t/development-setup-pycyphal-ubuntu/1844

As it relates to your question:

You can run the docs build/test using the following command (after the setup is done):

nox --session docs

The unit tests can also be run for each Python version seperately:

nox --session test-3.10

(You can also select a subset of tests to be run, see link above: Section 3, Step 5)

If you have any questions regarding the setup: feel free to ask them in the forum post! :)

pavel-kirienko · 2023-07-12T09:00:55Z

can't tell why the read the docs build failed.

Please check the logs:

They say:

/home/docs/checkouts/readthedocs.org/user_builds/pycyphal/checkouts/306/pycyphal/transport/can/media/socketcand/_socketcand.py:docstring of pycyphal.transport.can.media.socketcand._socketcand.SocketcandMedia:18: WARNING: Literal block ends without a blank line; unexpected unindent.
/home/docs/checkouts/readthedocs.org/user_builds/pycyphal/checkouts/306/pycyphal/transport/can/media/socketcand/_socketcand.py:docstring of pycyphal.transport.can.media.socketcand._socketcand.SocketcandMedia.__init__:15: WARNING: Unexpected indentation.
/home/docs/checkouts/readthedocs.org/user_builds/pycyphal/checkouts/306/pycyphal/transport/can/media/socketcand/_socketcand.py:docstring of pycyphal.transport.can.media.socketcand._socketcand.SocketcandMedia.__init__:16: WARNING: Block quote ends without a blank line; unexpected unindent.

pavel-kirienko

If I understand how it's built correctly, we no longer need the _construct_socketcand handler, the classic/FD differentiation (set the fd flag in the constructor when creating ThreadSafeBus), and the auxiliary types like PythonCANBusOptions and _InterfaceParameters. Removing these should simplify the implementation quite a bit.

Please format the code correctly using Black (more on this in CONTRIBUTING.md).

The old Yakut examples with --transport should be removed because this is no longer the recommended usage pattern.

This fixes the unnecessary pieces left from the pythonCANMedia class, such as the FD vs Classic dataclasses, Can Bus Options, and the constructor class. It also updates the way parameters are inputed to be all in the iface_name param like 'socketcand:can:host:port'

pycyphal/transport/can/media/socketcand/_socketcand.py

pavel-kirienko · 2023-07-14T18:55:30Z

Please try adding a simple integration test for this media implementation. You can make a new test case similar to the SocketCAN test that starts a local socketcand daemon and connects to it.

Added unit test and changed spec string to be parsed in _transport_factory fixed formatting issues that did not pass Black guidelines Removed unecessary params S Updated version of the socketcandMedia class. This fixes the unnecessary pieces left from the pythonCANMedia class, such as the FD vs Classic dataclasses, Can Bus Options, and the constructor class. It also updates the way parameters are inputed to be all in the iface_name param like 'socketcand:can:host:port' SocketcandMedia Media class added for socketcand library functionality. It runs on Python-CAN like the interfaces in the PythonCANMedia class, but because it has extra variables (host and port), it is not very clean to just add socketcand to PythonCANMedia.

…for output

pycyphal/transport/can/media/socketcand/_socketcand.py

tests/transport/can/media/_socketcand.py

pavel-kirienko · 2023-08-29T22:51:37Z

You should now be able to run the test suite by pushing a commit with the #test hashtag in the commit message.

Test environment now properly builds and breaks down socketcand Tests use rx_a and rx_b for proper communication

pavel-kirienko · 2023-08-30T00:54:07Z

Simply connect to any of the existing ones.

…

On Wed, Aug 30, 2023, 03:49 wiboticanders ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In tests/transport/can/media/_socketcand.py <#306 (comment)>: > + if sys.platform.startswith("linux"): + import resource # pylint: disable=import-error + + _logger.info("Limiting process memory usage to %.1f GiB", MEMORY_LIMIT / GIBIBYTE) + resource.setrlimit(resource.RLIMIT_AS, (MEMORY_LIMIT, MEMORY_LIMIT)) + + # Set up virtual SocketCAN interfaces. + execute("sudo", "modprobe", "can") + execute("sudo", "modprobe", "can_raw") + execute("sudo", "modprobe", "vcan") + execute("sudo", "ip", "link", "add", "dev", "vcan3", "type", "vcan", ensure_success=False) + execute("sudo", "ip", "link", "set", "vcan3", "mtu", 72) # Enable both Classic CAN and CAN FD. + execute("sudo", "ip", "link", "set", "up", "vcan3") Should I change it so that it adds one more vcan instance for mine to connect too? or simply connect to one of the ones previously instantiated — Reply to this email directly, view it on GitHub <#306 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFIZD4D4U43L6ITNJ6E23XX2EX7ANCNFSM6AAAAAA2GWWAAM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

pavel-kirienko · 2023-08-30T01:16:02Z

These steps are highly platform-dependent so they are better moved to the workflow file. Currently I can't even run the test on my PC because I'm using not an APT-based distro.

…

On Wed, Aug 30, 2023, 04:05 wiboticanders ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In tests/transport/can/media/_socketcand.py <#306 (comment)>: > + # build and install socketcand + execute("sudo", "apt-get", "install", "-y", "autoconf") + execute("git", "clone", "https://github.com/linux-can/socketcand.git") + execute("./autogen.sh", cwd="socketcand") + execute("./configure", cwd="socketcand") + execute("make", cwd="socketcand") + execute("sudo", "make", "install", cwd="socketcand") I might be mistaken but for local testing GH workflow isn't used, and nox is. So I think it would make more sense for me to put it in the nox file so that it can be run. I also thought it may be possible to put it the conftest.py file. Do you have a preference on any of these — Reply to this email directly, view it on GitHub <#306 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFIZBUNMLGQRFXO4RYEU3XX2GV5ANCNFSM6AAAAAA2GWWAAM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

wiboticanders · 2023-08-30T01:21:07Z

So possibly something like this within workflow: name: Setup socketcand run: | sudo apt-get install -y autoconf git clone https://github.com/linux-can/socketcand.git cd socketcand ./autogen.sh ./configure make sudo make install shell: bash And then within the actual unit test I can run the daemon and take it down. On Tue, Aug 29, 2023 at 6:16 PM Pavel Kirienko ***@***.***> wrote:

…

These steps are highly platform-dependent so they are better moved to the workflow file. Currently I can't even run the test on my PC because I'm using not an APT-based distro. On Wed, Aug 30, 2023, 04:05 wiboticanders ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In tests/transport/can/media/_socketcand.py > <#306 (comment)>: > > > + # build and install socketcand > + execute("sudo", "apt-get", "install", "-y", "autoconf") > + execute("git", "clone", "https://github.com/linux-can/socketcand.git") > + execute("./autogen.sh", cwd="socketcand") > + execute("./configure", cwd="socketcand") > + execute("make", cwd="socketcand") > + execute("sudo", "make", "install", cwd="socketcand") > > I might be mistaken but for local testing GH workflow isn't used, and nox > is. So I think it would make more sense for me to put it in the nox file so > that it can be run. I also thought it may be possible to put it the > conftest.py file. Do you have a preference on any of these > > — > Reply to this email directly, view it on GitHub > <#306 (comment)>, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AAZFIZBUNMLGQRFXO4RYEU3XX2GV5ANCNFSM6AAAAAA2GWWAAM> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#306 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BA63ESEBHSL4EPSC2KR4FI3XX2H5ZANCNFSM6AAAAAA2GWWAAM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

pavel-kirienko · 2023-08-30T09:33:40Z

So possibly something like this within workflow

Yes!

And then within the actual unit test I can run the daemon and take it down

Also modified docs of SocketcandMedia, and updated socketcandMedia test to only start the daemon instead of building socketcand, setting up a vcan, and then running the daemon, now socketcand is installed in workflow and we use an existing vcan set up previously. I also added some error handling for running the socketcand daemon and made sure to kill the process correctly

wiboticanders · 2023-08-30T17:41:15Z

I put #test in my commit message, but it seems like it's still skipping the test and release tests. Also, assuming all works correctly in the workflow file, this should be close to a finalized and working version

pavel-kirienko · 2023-08-31T12:06:41Z

I think it should work now.

coveralls · 2023-08-31T19:14:34Z

coverage: 95.212% (-0.4%) from 95.569% when pulling 00baae9 on wiboticanders:master into 1eefb91 on OpenCyphal:master.

Co-authored-by: Pavel Kirienko <pavel.kirienko@gmail.com>

wiboticanders · 2023-09-01T00:17:02Z

Commit #[00baae9] revealed an error in the breakdown when the test fails. I was going through suggestions on the github site and not locally, and I thought it was all changes to the docs, so I committed the suggestions. One slipped past me that was a change to the interface return, which in turn broke the unit test because the returns did not match anymore. The problem stems from the way I collecting stderr and stdout after the yield, the communicate function never ended. So, that commit will not stop testing until whatever timeouts you have in place stop it. My apologies for overlooking that issue and pushing it, but I can't seem to stop it with my permissions so it has been running for a few hours.

In the previous commit it was mostly doc updates, but one changed the socketcandmedia interface_name return. I forgot to update the unit test assertions to match this, so in the CI testing it resulted in a never ending loop. The issue stemmed from the way Popen gets stderr and stdout with the communicate method, but only after the yield statement. It does not respond to the kill() and instead runs forever, so I replaced it with just a kill. Although you lose a little bit of info, the socketcand instance gets checked before the unit test is done in the fixture, and from there the daemon is very robust and will not stop unless you force kill it or terminate it. So I think this is the best solution, you still get output from errors within the unit test as well.

pavel-kirienko

Please bump the minor version number in _version.py and drop a line in the changelog.

.github/workflows/test-and-release.yml

tests/transport/can/media/_socketcand.py

pycyphal/transport/can/media/socketcand/_socketcand.py

tests/transport/can/media/_socketcand.py

wiboticanders · 2023-09-07T20:10:10Z

So new version should be 1.16.0?

Also made small tweaks based on code change requests, removed some redundacy, slightly changed docs.

wiboticanders added 18 commits July 6, 2023 10:52

added new functionality within pythoncanmedia to work with the socket…

e79c885

…cand interface

test

e0e4026

Added the typing.Optional to host and port

12a6b12

in init function

changed the socketcand interface variables from

6f54abe

the old implimentation to a new one

Cleaned up some junk left in from old commits,

0cf6e65

added another error check for socketcand and CAN FD at same time

test

9063059

Changed placement of new variables

720166e

Changed placement of new variables

aeb722e

Version w/o socketcand data class, otherwise same

98b1dbe

test

f576f22

Fixed small variable missname, added documentation

c57c5ed

to file as well

added more documentation

4a30f62

changed defualt port

a1e8ffc

Changed host and port vars to be in the superclass

a7e65e7

_InterfaceParameters instead of separately in the FD and classic subclasses

Merge branch 'noNewDataClass'

196876f

test

d9f8267

Revert "test"

9b48f1d

This reverts commit d9f8267.

SocketcandMedia Media class added for socketcand

d08509a

library functionality. It runs on Python-CAN like the interfaces in the PythonCANMedia class, but because it has extra variables (host and port), it is not very clean to just add socketcand to PythonCANMedia.

pavel-kirienko requested changes Jul 12, 2023

View reviewed changes

pavel-kirienko mentioned this pull request Jul 13, 2023

Updated version of the socketcandMedia class. #308

Closed

pavel-kirienko requested changes Jul 14, 2023

View reviewed changes

wiboticanders added 4 commits July 19, 2023 13:14

Merge branch 'master' of https://github.com/wiboticanders/pycyphal

b74bb9b

Merge branch 'experimental'

53036ca

Going back to original _pythoncan version

ac1fac9

wiboticanders added 2 commits August 28, 2023 18:03

Fixed socketcand command to be run in the background without waiting …

fd828db

…for output

Updated test frames sent over socketcand

c0ecd0f

pavel-kirienko requested changes Aug 29, 2023

View reviewed changes

Merge branch 'master' into master

f25c069

Updated test environment and tests

ccb9ef2

Test environment now properly builds and breaks down socketcand Tests use rx_a and rx_b for proper communication

Update test-and-release.yml

e379145

pavel-kirienko and others added 3 commits August 31, 2023 15:08

Update test-and-release.yml

fb3b99d

Fixed mypy typing errors

6ad2e1e

fixed black checkstyle

67ac831

wiboticanders and others added 2 commits August 31, 2023 12:56

Fixed socketcand error collecting and catching

585c7c1

Apply suggestions from code review

00baae9

Co-authored-by: Pavel Kirienko <pavel.kirienko@gmail.com>

wiboticanders requested a review from pavel-kirienko September 7, 2023 16:56

pavel-kirienko requested changes Sep 7, 2023

View reviewed changes

Updated minor version number and changelog

29917e3

Also made small tweaks based on code change requests, removed some redundacy, slightly changed docs.

wiboticanders requested a review from pavel-kirienko September 8, 2023 16:48

Update _socketcand.py

87c7a49

pavel-kirienko approved these changes Sep 8, 2023

View reviewed changes

pavel-kirienko merged commit 4846761 into OpenCyphal:master Sep 8, 2023
8 of 9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SocketcandMedia Media class added for socketcand functionality #306

SocketcandMedia Media class added for socketcand functionality #306

wiboticanders commented Jul 11, 2023

wiboticanders commented Jul 11, 2023

maksimdrachov commented Jul 12, 2023

pavel-kirienko commented Jul 12, 2023

pavel-kirienko left a comment •

edited

Loading

pavel-kirienko commented Jul 14, 2023 •

edited

Loading

pavel-kirienko commented Aug 29, 2023

pavel-kirienko commented Aug 30, 2023 via email

pavel-kirienko commented Aug 30, 2023 via email

wiboticanders commented Aug 30, 2023 via email

pavel-kirienko commented Aug 30, 2023

wiboticanders commented Aug 30, 2023

pavel-kirienko commented Aug 31, 2023 •

edited

Loading

coveralls commented Aug 31, 2023 •

edited

Loading

wiboticanders commented Sep 1, 2023 •

edited

Loading

pavel-kirienko left a comment

wiboticanders commented Sep 7, 2023

SocketcandMedia Media class added for socketcand functionality #306

SocketcandMedia Media class added for socketcand functionality #306

Conversation

wiboticanders commented Jul 11, 2023

wiboticanders commented Jul 11, 2023

maksimdrachov commented Jul 12, 2023

pavel-kirienko commented Jul 12, 2023

pavel-kirienko left a comment • edited Loading

Choose a reason for hiding this comment

pavel-kirienko commented Jul 14, 2023 • edited Loading

pavel-kirienko commented Aug 29, 2023

pavel-kirienko commented Aug 30, 2023 via email

pavel-kirienko commented Aug 30, 2023 via email

wiboticanders commented Aug 30, 2023 via email

pavel-kirienko commented Aug 30, 2023

wiboticanders commented Aug 30, 2023

pavel-kirienko commented Aug 31, 2023 • edited Loading

coveralls commented Aug 31, 2023 • edited Loading

wiboticanders commented Sep 1, 2023 • edited Loading

pavel-kirienko left a comment

Choose a reason for hiding this comment

wiboticanders commented Sep 7, 2023

pavel-kirienko left a comment •

edited

Loading

pavel-kirienko commented Jul 14, 2023 •

edited

Loading

pavel-kirienko commented Aug 31, 2023 •

edited

Loading

coveralls commented Aug 31, 2023 •

edited

Loading

wiboticanders commented Sep 1, 2023 •

edited

Loading