Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debugger Transport APIs and Zephyr port #2247

Closed

Conversation

jimmy-huang
Copy link
Contributor

This is patch to tackle issue #2229, basically we want the ability to have debugger support different platforms and different transports.

The current debugger implementation only supports BSD sockets, and it's not portable to different OSes that have different platform APIs. Besides, other platforms may want to use another transport layer like USB or Bluetooth instead of TCP sockets. The goal is to create a generic transport interface that different platforms can be used, and provide the actual implementation for running on their devices. This patch provides a struct that defines the function signature that each transport can be used to register their own implementation of the socket APIs using the jerry_port_xxx interface and a Zephyr port is provided as a sample.

This is a WIP patch and a reference implementation, but I would like use this as a starting point for discussion on how to improve the APIs from here, would like to have feedbacks from @zherczeg and @martijnthe and JerryScript team after reviewing them.

@martijnthe
Copy link
Contributor

martijnthe commented Mar 21, 2018

@jimmy-huang some builds are failing (click "Details" below to learn more)

* The default transport layer that uses TCP socket
* other transports can be added like usb, bluetooth
*/
jerry_debugger_transport_t *jerry_debugger_init_socket_transport(uint16_t tcp_port);
Copy link
Contributor

@martijnthe martijnthe Mar 21, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be in jerryscript-port.h?
It's specific to the socket transport implementation, no?

[EDIT] Ah, I guess you use the same function signature in the zephyr implementation as well.
Thinking about it again, it makes sense.

*/
jerry_debugger_transport_t*
jerry_debugger_init_socket_transport(uint16_t tcp_port) /**< server port number */
{
Copy link
Contributor

@martijnthe martijnthe Mar 21, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add an assert to make sure it's only called once (since the socket_transport state is global/singleton)?

Copy link
Contributor Author

@jimmy-huang jimmy-huang Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this only intended to be called once? if you call debugger_cleanup() and then restart the the cycle with debugger_init(), and you would expect to init the transport() again? or at least okay to call it as long as it's a singleton.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I see, ok, I guess it's ok.

There's nothing precluding one from creating a dynamically created transport (and having a transport-specific cleanup function to free/destroy the transport), so I'm happy.

@jimmy-huang jimmy-huang force-pushed the debugger-port branch 6 times, most recently from 8bb9dab to 24785b4 Compare March 22, 2018 00:53

#include "jerryscript-port.h"

#include "jrt.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is used from this header?
It's a private/internal header.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martijnthe i was trying to fix the travis build erros, because i have to call JERRY_UNUSED on the unused port parameter when debugger is ifdef'ed out, but JERRY_UNUSED is not defined, but i think i don't need to call it in the zephyr port, only in the default-socket.c?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martijnthe btw, I am still fixing some of these errors, i got around a bunch yesterday, but it turned out pulling in unistd.h for ssize_t is problematic on other platforms, so I think i will change it to use size_t only on the bytes send/received values, i mean it was initially copied from the socket calls, i think making it just unsigned is okay since we return the error values anyway, so no need to check -1.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds reasonable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JERRY_UNUSED

Hmm I guess it's not part of the "API"

Maybe just use (void) variable_that_is_unused?

@martijnthe
Copy link
Contributor

@zherczeg could you take a look at the design?

@jimmy-huang jimmy-huang force-pushed the debugger-port branch 3 times, most recently from 24ac32a to 2609b6a Compare March 22, 2018 20:35
Copy link
Member

@zherczeg zherczeg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The general direction is ok, but I feel the transport and transport below websocket is somehow mixed. General functions should be moved to jerry_debugger.c from jerry_debugger_ws.c.

@@ -93,13 +93,18 @@ jerry_debugger_stop_at_breakpoint (bool enable_stop_at_breakpoint) /**< enable/d
* Debugger server initialization. Must be called after jerry_init.
*/
void
jerry_debugger_init (uint16_t port) /**< server port number */
jerry_debugger_init (jerry_debugger_transport_t *transport_p) /**< transport */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this port specific definition part of the public API?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean the jerry_debugger_transport_t type? Yes, this is intended and part of the jerry_port_ api where each platform have to provide their own definition, hence why it was forward declared, so it can build.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately "it can build" is not a strong enough reason for something to be part of the public API.

While I am ok with having a structure for communication, the public API part needs to be independent of everything: no jerry port dependency, no compiler define dependency, etc.

I think the first question is: do we want multiple transport layer support, or the recv/send/close could be provided by jerry port.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zherczeg When I originally started the the port, it was a simpler patch to just port the socket APIs, but I think @martijnthe really like to support multiple transports, so we kinds modified it to take a struct of handlers. I mean we can submit a simpler patch to ports only socket calls and then a separate patch later to rework the APIs to support multiple transports, I think it's up to you and the JerryScript maintainers to make this kind of decision. Honestly, my goal is to at least get the websockets working on Zephyr as part of this effort. If we move the the struct definitions and APIs to from jerryscript-port to jerryscript-debugger.h, would that work for you?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this port specific definition part of the public API?

The definition should NOT be part of the public API, only the forward declaration should be. That way there is no dependency. (Until one actually want to use the debugger feature, in which case one has to pick an implementation which would contain the definition of the struct as well.)

I think the first question is: do we want multiple transport layer support

Yes, at least in our project we want to mix and match between 3 transports at run-time.

IMHO, a stronger reason for wanting a more dynamic transport interface as opposed to extending the "port surface" with static functions for the transport, is that static functions would make combining, composing and re-using transport implementations harder.

For example in our case, we want USB, BLE and a "pseudo transport" that just gathers the debug info (i.e. for printing human consumable exception traces). Imagine that someone already wrote each of these transports separately. Each transport would define the same set of functions/symbols. If I want to create a new transport that combines/chains/... them, I'd have to rename each function to avoid conflicts. Even if a simple "switch" between 2 transports would involve renaming each function.

@@ -17,6 +17,7 @@
#define JERRYSCRIPT_DEBUGGER_H

#include <stdbool.h>
#include "jerryscript-port.h"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely not good here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can change the init() to take a void*, thus avoid pulling in jerryscript-port.h

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or repeat the forward declaration?
Or move the forward declaration into an API header?
@zherczeg What's the best do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or repeat the forward declaration?
Or move the forward declaration into an API header and include that?
@zolWhat's the best do you think?

@@ -65,24 +60,20 @@ typedef struct
* Close the socket connection to the client.
*/
static void
jerry_debugger_close_connection_tcp (bool log_error) /**< log error */
jerry_debugger_close_connection_transport (void)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a plan to log connection error in the future?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, we can just change to all the handlers Apis to take the same log_error parameter

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like only this function has such an option.

if (!jerry_debugger_send_tcp ((const uint8_t *) text_p, strlen (text_p))
|| !jerry_debugger_send_tcp (request_buffer_p + sha1_length + 1, 27))
if (!jerry_debugger_send_transport ((const uint8_t *) text_p, strlen (text_p))
|| !jerry_debugger_send_transport (request_buffer_p + sha1_length + 1, 27))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite websocket specific. As far as I know websockets is available only through normal sockets.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i could leave it with just _tcp if it's ok, @martijnthe ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know websockets is available only through normal sockets.

At this point I'm regarding the websocket framing and handshaking to be part of the JrS debugger protocol as you yourself suggested a year ago: #1557 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was a long time ago when I worked on this. But it seems a reasonable idea. Probably the handshake could be an optional feature.

*/
typedef struct jerry_debugger_transport_t jerry_debugger_transport_t;
typedef struct jerry_debugger_config_t jerry_debugger_config_t;
typedef struct jerry_debugger_conn_t jerry_debugger_conn_t;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to forward declare these functions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these needs to be forward declared because it's up to the jerry_port layers to provide the definitions for these, it will run into compile errors if debugger is not enabled.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The struct definitions itself is up to the implementation of the transport. Initially, @jimmy-huang had used void *, but I suggested to use forward declarations. That way the compiler will still help with ensuring the correct types are used.

uint8_t debugger_message_delay; /**< call receive message when reaches zero */
int debugger_connection; /**< holds the file descriptor of the socket communication */
jerry_debugger_transport_t *debugger_transport_p; /**< holds the pointer to the debugger transport */
jerry_debugger_conn_t *debugger_connection_p; /**< holds the pointer to the debugger connection */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to separate these two?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can probably keep just the transport obj, and have the connection be part of the transport, @martijnthe ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think so.

I don't see a use case for being able to distinguish the transport and the connection.

I guess in a multi-Jerry-context one could have multiple connections, sharing the same transport implementation. But one could work around that by just creating separate jerry_debugger_transport_ts instead I guess.

@zherczeg
Copy link
Member

JERRY_DEBUGGER_INIT_SEND_MESSAGE
JERRY_DEBUGGER_SET_SEND_MESSAGE_SIZE_FROM_TYPE
JERRY_DEBUGGER_SET_SEND_MESSAGE_SIZE

jerry_debugger_accept_connection
jerry_debugger_close_connection
jerry_debugger_send
jerry_debugger_receive

jerry_debugger_send_header_t

It seems these types/functions are used by the debugger. If we want to stick to the websocket package format, perhaps the first step would be to move certain of these types into the main debugger, reducing the content of debugger-ws.h as much as possible. If not, the transport system must provide a way to provide these declarations.

@zherczeg
Copy link
Member

By the way the send / receive already working in different ways, since the send aware of the low-level websocket header format while it is stripped from the receive part. Perhaps the JERRY_DEBUGGER_INIT_SEND_MESSAGE could be removed first, and moved internally to jerry_debugger_send in debugger-ws.c. One less macro to worry about.

@martijnthe
Copy link
Contributor

Re. keeping websocket packet framing:

I'm OK with either, keeping it or moving it out. I do think that it's a tad bit wasteful / annoying if the transport itself also (inherently) takes care of packet framing (I'm thinking about BLE for example where every byte saved in the air is important).
It was just your past comment that made me take the initial position of "OK the websocket framing is part of the JrS debug protocol".

If we decide to move it out, then I agree, there's quite a bit of rework needed to separate the WS related stuff out and into the transport side of the fence.

If we decide to keep WS and regard it as part of the JrS debug protocol, I'm not sure why it's needed to refactor the debugger-ws.c/debugger.c split in order to progress with the transport layer abstraction. IMHO, cleaning up those files would be better off as a separate PR.

@jimmy-huang thoughts?

@zherczeg So it sounds like a decision needs to be made about "websocket framing is part of JrS debug protocol": yes or no. Any other people that need to chime in to help make that decision?

@jimmy-huang
Copy link
Contributor Author

@martijnthe Yeah, personally, i rather keep the WS at least for now, because initially I think most of the platforms will just re-use the same WS protocol, and like you said, it's a lot of work for each port to add these WS. I think the WS and both the TCP socket can be consider as a transport, so for now if we just move the TCP logic into a tcp transport, then later we can then move the WS into a separate transport as well. But I am okay with either as well.

@zherczeg
Copy link
Member

zherczeg commented Mar 28, 2018

I would first remove as much macros / types from debugger-ws.h as possible. Probably in separate PR-s first. JERRY_DEBUGGER_INIT_SEND_MESSAGE could go away easily (send should write this byte). The JERRY_DEBUGGER_SET_SEND_MESSAGE_SIZE_FROM_TYPE and
JERRY_DEBUGGER_SET_SEND_MESSAGE_SIZE macros probably could go away by adding a size parameter to send. Basically the transport protocol should specify how much (maximum) bytes should be reserved for the header in the send buffer. This should be a constant of course (if less bytes are needed for the actual message, just internally adjust the start before sending the packet). The size passed to send excludes this amount of course.

@jimmy-huang
Copy link
Contributor Author

@zherczeg where does jerry_debugger_send_header_t go? Should it be moved to debugger.h? or it's a internal thing that's in debugger-ws.c? I am guessing the former?

@jimmy-huang
Copy link
Contributor Author

@zherczeg I submitted another patch #2262 to remove some of these macros. Is the goal here to make the debugger-ws.c as the default transport and remove all the macros and structurs in debuggr-ws.h. What about these structs?

jerry_debugger_send_header_t
jerry_debugger_receive_uint8_data_part_t
jerry_debugger_uint8_data_t

@martijnthe
Copy link
Contributor

martijnthe commented Mar 30, 2018

Basically the transport protocol should specify how much (maximum) bytes should be reserved for the header in the send buffer. This should be a constant of course (if less bytes are needed for the actual message, just internally adjust the start before sending the packet). The size passed to send excludes this amount of course.

@jimmy-huang @zherczeg Alternative idea: instead of letting the transport specify the max header size, how about if the debugger instead would ask the transport for a pointer to a send buffer? The transport itself would take care of reserving space for a header. In case sending happens async, it could also save a copy to another buffer + lower the peak memory usage a little (perhaps not relevant though). Downside is that getting a buffer would become a function call instead of just getting a pointer + offset from the context, which in the simplest of projects would be at a fixed memory location. OTOH, there are only ~10 or so places, so in absolute .text size footprint it may be acceptable.

I'm not feeling strongly about either, but wanted to share the idea.

@zherczeg
Copy link
Member

@jimmy-huang I would not use .something fields. Simply create some functions such as jerry_debugger_set_transmit_sizes with 3 arguments. Such as @martijnthe said, we should not be able to switch transport after it is set for a jerry instance. I would like to some suggestions how could we remove jcontext.h dependency from debugger-ws.c and do it in a separate patch.

@martijnthe
Copy link
Contributor

@zherczeg

I would like to some suggestions how could we remove jcontext.h dependency from debugger-ws.c and do it in a separate patch.

I'm not sure if I understand. This PR removes the jcontext.h dependency from debugger-ws.c already? Would you like to see this PR split in 2 separate ones? If so, what should be left out?

Please clarify. Thank you.

@jimmy-huang
Copy link
Contributor Author

@zherczeg @martijnthe Updated with jerry_debugger_set_transmit_sizes function

@jimmy-huang
Copy link
Contributor Author

@zherczeg Can take take a another look at the patch?

@jimmy-huang jimmy-huang force-pushed the debugger-port branch 2 times, most recently from 2aa9821 to bf2c3da Compare April 25, 2018 01:05
@jimmy-huang jimmy-huang changed the title [WIP] - Debugger Transport APIs and Zephyr port Debugger Transport APIs and Zephyr port May 7, 2018
@jimmy-huang
Copy link
Contributor Author

@zherczeg can you take a look at this patch to see if there's something else I should fix?

@zherczeg
Copy link
Member

zherczeg commented May 8, 2018

The problem is not with the patch. The problem lies in nicely restructuring the debugger interface. I spent some hours to decouple the ws part from jcontext, but I was not satisfied with the results. It worked, but the code was ugly. Unfortunately this must be the first step. So this is the challenge here. If you have time, you can try it.

JerryScript-DCO-1.0-Signed-off-by: Jimmy Huang jimmy.huang@intel.com
This patch adds the transport interfaces for the debugger and allows it to
be ported to different platforms.

JerryScript-DCO-1.0-Signed-off-by: Jimmy Huang jimmy.huang@intel.com
Move all generic debugger logic into debugger.c and
remove as many calls to internal APIs as possible.

JerryScript-DCO-1.0-Signed-off-by: Jimmy Huang jimmy.huang@intel.com
JerryScript-DCO-1.0-Signed-off-by: Jimmy Huang jimmy.huang@intel.com
JerryScript-DCO-1.0-Signed-off-by: Jimmy Huang jimmy.huang@intel.com
@jimmy-huang
Copy link
Contributor Author

@zherczeg Ok, that's essentially what I am doing with one of the commits, so I separate it out as a separate patch, maybe this is what you are looking?

https://github.com/jimmy-huang/jerryscript/tree/refactor-ws

@zherczeg
Copy link
Member

Not exactly. I would like to be able to chain this at the top of other interfaces, e.g. tcp. Maybe move it to some jerry extension or port. I am not sure.

@jimmy-huang
Copy link
Contributor Author

Sorry, I don't quite understand, maybe if you can show me some of the work you are doing? I guess maybe it makes more sense for you to actually do the refactoring, since you guys are more familiar with the APIs? I'd be happy to help but our resource is kind limited so I am not able to put too much time into this, but I would at least want to get to a state where at least someone can easily port this interface to Zephyr or other platforms.

Copy link
Member

@akosthekiss akosthekiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for chiming in late to this conversation but something doesn't feel right about the forward-declared types in the debugger API and/or the port API. I may not see the whole picture but I don't really grasp how a forward declared-only struct type would work with the ports. If the ports don't see the whole type, why do they need it at all? If the type is internal to the debugger implementation, why shall it be published? Any clarification is appreciated.

@@ -38,18 +38,12 @@ typedef struct
/**
* Byte data for evaluating expressions and receiving client source.
*/
typedef struct
typedef struct jerry_debugger_uint8_data_t
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change seems to be unjustified. Current code base adds names to those structs only that are recursive. Here, it's not the case.

*/

/* forward declaration */
struct jerry_debugger_uint8_data_t;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This forward declaration also seems to be unnecessary, especially in this public header. The type is only used in sources internal to jerry-core. If I see it right.

@jimmy-huang
Copy link
Contributor Author

@zherczeg @akosthekiss Thanks for the feedback, been out for a while and just got back from vacation. We actually have a work-around building debugger on top of Zephyr's native posix network stack, since they are almost identical. So in our project, we have this custom header that get injected when debugger is turned on. I can probably submit a patch to the zephyr target as well, is this something you guys want? Since at this point, we don't need to support other transports besides tcp. This will not have any changes on the API too.

https://github.com/intel/zephyr.js/blob/master/src/jerry-port/arpa/inet.h

@zherczeg
Copy link
Member

zherczeg commented Jun 1, 2018

Unfortunately I have no time to work on this at the moment. The whole debugger interface needs a major redesign to satisfy all requirements. I would recommend to use your workaround until we can do this.

@zherczeg
Copy link
Member

https://github.com/zherczeg/jerryscript/tree/debugger_transport

Something to start with. It is far from compiling though. This will take a long time to finish.

@zherczeg
Copy link
Member

Please check #2421

@zherczeg
Copy link
Member

Hi @jimmy-huang, after #2426 lands you can freely add your low-level transmission protocols to JerryScript. Please check the patch if you have time.

@jimmy-huang
Copy link
Contributor Author

@zherczeg Yeah, thanks for the heads up, I am actually ramping up on another project atm, but I'll try to look at this later when I have time, if you think i should close this PR first, I am okay with it and I'll re-submit a pr later.

@jimmy-huang
Copy link
Contributor Author

closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants