Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 565 lines (404 sloc) 22.592 kb
fccc685 Initial open-source release
MLstate authored
1 -*-text-*-
2
3 MLIDL
4 =====
5
6 This is a program which takes as input a file containing a (limited) set of
7 OCaml type definitions plus some annotations and generates code in OCaml and OPA
8 to transmit these values between networked instances. In theory, the encoded
9 types should be compatible with the current JSON types and could be used to send
10 data to javascript endpoints but this has not been tested.
11
12 The mlidl program
13 =================
14
15 Recognised types
16 ----------------
17
18 This takes as input the following subset of OCaml types, defined in pseudo-ML
19 types:
20
21 <code>
22 open MLIDL_Module
23
24 external external_ty : Mod.external_type = "<complex set of implementation functions>"
25
26 type ty =
27 unit
28 | int
29 | float
30 | bool
31 | string
32 | ty option
33 | ty list
34 | (ty * ty * ... )
35 | { lab1:ty; lab2:ty; ... }
36 | (Cons1 of ty | Cons2 of ty | Cons3)
37 | MLIDL_Module.ty
38 | external_ty
39 </code>
40
41 Note that there are no:
42
43 - Type variables: 'a ty
44 - References: ty ref
45 - Functional types: ty -> ty -> ...
46 - Type labels: ~label:ty
47
48 Although some support is planned for a simplified form of type variables which
49 will require a complex set of definitions as used by the external type syntax.
50
51 You can import types from other IDL files with the syntax indicated above. You
52 have to "open" the name of the generated module by the included IDL file and
53 then prefix *all* of the types you import from other files.
54
55 There is also provision for importing externally-defined types from OCaml
56 modules but the syntax for this is highly complex. Simply put, you have to do
57 all the work done for you by the IDL code generator and indicate to the IDL
58 function names which are the equivalent of its own generated functions for these
59 types. This is a lot of work so you should maybe consider using the IDL program
60 to generate these types and then use the import mechanism if you have access to
61 the sources of these types. One point, however, is that this mechanism allows
62 you to define a completely separate type representation between the OCaml and
63 OPA types. See the "External Types" section below for more details.
64
65 Basic functionality generated
66 -----------------------------
67
68 The types are translated more-or-less literally into OCaml types on the ML side
69 and equivalent types on the OPA side. The OCaml output code includes an
70 interface file. The following functions are generated:
71
72 String input/output
73 ...................
74
75 Firstly, for OCaml this functionality is wrapped in functors (MakeInput and
76 MakeOutput) where the input and output functions are abstract. However, for
77 convenience, an instantiation of these modules is provided generating the
78 functions:
79
80 - <type>_of_string
81 - string_of_<type>
82
83 for each <type> in the IDL file. These are the functions used by the
84 network layer code. If you wish to output to Buffer.t or whatever then
85 instantiate the functors.
86
87 For OPA the output functions are called:
88
89 - output_<type>
90
91 and are wrapped in a module called STR_<name>, where <name> is the base name
92 of the IDL file. There is currently no input_<type> functionality for
93 native OPA (due to the slowness of OPA parsers).
94
95 For BSL output, the <type>_of_string and string_of_<type> functions are made
96 available to OPA where the types remain external OCaml types. Note,
97 however, that the wrap_<type> and unwrap_<type> functions used by these
98 routines are also made available to OPA.
99
100 Generation of these functions can be prevented with the option:
101
102 --string-functions false
103
104 JSON input/output
105 .................
106
107 These are functions to convert the types into JsonTypes types in OCaml and
108 to RPC.Json.json types for OPA:
109
110 - tojson_<type>
111 - fromjson_<type>
112
113 These functions are available for both OCaml and Native OPA. The OPA
114 functions are included in a module called "JSON_<name>".
115
116 Currently, the BSL generated code does not include JSON conversion.
117
118 There used to be support for low-level JSON types in OPA (OCaml JsonTypes as
119 external types in the OPA file) but this has fallen out of date since the
120 Json types were updated. It could be resurrected on request.
121
122 You can individually defeat the tojson_<type> and fromjson_<type> functions
123 with:
124
125 --tojson-functions false
126 --fromjson-functions false
127
128 Create types
129 ............
130
131 There is primitive support for generating types from constant types in the
132 constructors:
133
134 IDL file: type date = { year : int; month : int; day : int; }
135
136 MLI file: val create_date : int -> int -> int -> date
137
138 Application: let date = create_date 1961 8 24;;
139
140 These can be switched off by:
141
142 --create-functions false
143
144 Network layer code generated
145 ----------------------------
146
147 The IDL code generator also understands a small set of annotations (disguised as
148 OCaml "val" statements) which indicate that it should generate code for the
149 network layer communications which use the basic conversion functions:
150
151 <code>
152 val sendreceive_profile : profile -> profile
153 val send_date : hare
154 val receive_gender : gender
155 val protocol_cat : cat -> string
156 val responder_dog : dog -> string
157 </code>
158
159 Don't be fooled by the ML-like syntax of these annotations, there are no such
160 values actually generated by the IDL code generator.
161
162 The most important of these is the "responder" annotation which generates all
163 the code needed to implement a server/client combination between two (Hlnet)
164 endpoints. The other routines provide low-level network operations which may be
165 useful where the responder pattern is not what is required. A brief summary of
166 these operations:
167
168 - "val sendreceive_<name> : <type1> -> <type2>"
169
170 This is an Hlnet.sendreceive call where the types are encoded according to
171 the encoding defined in the IDL file (currently there is only one encoding).
172
173 - "val send_<name> : <type>"
174
175 Generates an Hlnet.send call with the encoded type.
176
177 - "val receive_<name> : <type>"
178
179 Generates an Hlnet.receive call with the encoded type.
180
181 - "val protocol_<name> : <type1> -> <type2>"
182
183 Generates an Hlnet.Aux.easy_protocol value with query type <type1> and
184 response type <type2>. Note that you can set the name and version number for
185 the protocol by annotations in the IDL file. Note also that if you set
186 either type to "string" then no type encoding is applied to that type.
187
188 - "val responder_<name> : <type1> -> <type2>"
189
190 This simple statement generates all the apparatus needed to implement a
191 server/client pair using the indicated types. See the "Responders" section
192 below for details.
193
194 Command line and in-built arguments
195 -----------------------------------
196
197 The output of the IDL file can be controlled by either command-line arguments or
198 with annotations built into the IDL files themselves.
199
200 One important point is that the in-built parameters take precedence. If you
201 wish to control the output from the command line, the in-built argument
202 equivalents must not be present. Here are the command line arguments:
203
204 -output-suffix <string> Output suffix (default: "types").
205
206 The generated files (ml, mli, opa, bsl) will all have this suffix appended
207 to the base name of the IDL file (eg. test.mlidl -> testtypes.ml).
208
209 -bsl-prefix <string> Bsl prefix (default: "bsl").
210
211 This prefix will be prepended to the base name of generated BSL files. This
212 is to prevent overwriting the ml files of the same name. Note that if your
213 code is not destined for the OPA BSL mechanism, a more suitable prefix might
214 be "plugin".
215
216 -ocaml-wrap-options <bool> Wrap option around OCaml input (default: "true").
217 -opa-wrap-options <bool> Wrap option around Opa input (default: "true").
218
219 Redundant, do not use.
220
221 -native-parser <bool> Use native parser instead of TRX wrappers (default: "true").
222
223 Redundant, do not use.
224
225 -hlnet-logging <bool> Add logger statements to Hlnet wrappers (default: "false").
226
227 Network-layer code can optionally generate debug logging message, this flag
228 switches this functionality on and off.
229
230 -logger-function <string> Logger function (default: "Logger.log").
231
232 The logger function to use for the --hlnet-logging feature.
233
234 -protocol-version <int> Protocol version number (default: 1).
235
236 The protocol number to install in any protocols generated for the network
237 layer.
238
239 -default-port <int> Default port number (default: 49152).
240
241 The default port to be built into the network layer endpoint description.
242
243 -default-addr <string> Default inet address (default: "Unix.inet_addr_loopback").
244
245 The default address for the endpoint.
246
247 -create-functions <bool> Output create value from type functions (default: true).
248
249 Add the create functions to the output.
250
251 -tojson-functions <bool> Output type to json functions (default: true).
252 -fromjson-functions <bool> Output type from json functions (default: true).
253
254 Enable/disable JSON output/input functions.
255
256 -string-functions <bool> Output type to/from string functions (default: true).
257
258 Enable/disable to/from string functions.
259
260 -bsl-file <bool> Output BSL file for OCaml functions (default: true).
261
262 Generate BSL wrapper code. Note: you need -no-ocaml to be false since only
263 the wrappers are generated, not the referenced OCaml code.
264
265 -no-ocaml <bool> Don't generate OCaml output (default: false).
266
267 Do not generate OCaml output.
268
269 -no-opa <bool> Don't generate OPA output (default: false).
270
271 Do not generate native OPA output.
272
273 The in-built option have the following syntax:
274
275 let module_name = "test2"
276 let protocol_version = 2
277 let verbose = true
278
279 The value must be of the correct type. Mostly these are exactly the same as the
280 command-line arguments:
281
282 In-built argument Type Command-line option
283 ----------------- ------ -------------------
284 module_name String <none>
285 output_suffix String --output-suffix
286 bsl_prefix String --bsl-prefix
287 encoding_number Int --encoding-number
288 ocaml_wrap_opt Bool --ocaml-wrap-opt
289 opa_wrap_opt Bool --opa-wrap-opt
290 native_parser Bool --native-parser
291 hlnet_logging Bool --hlnet-logging
292 logger_function String --logger-function
293 protocol_version Int --protocol-version
294 default_port Int --default-port
295 default_addr String --default-addr
296 create_functions Bool --create-functions
297 tojson_functions Bool --tojson-functions
298 fromjson_functions Bool --fromjson-functions
299 string_functions Bool --string-functions
300 bsl_file Bool --bsl-file
301 no_ocaml Bool --no-ocaml
302 no_opa Bool --no-opa
303 verbose Bool --v
304 debug Bool --g
305
306 The "module_name" argument does not appear on the command line since the name of
307 the output module will apply to each IDL file, it would not make sense to set
308 the same module name for all included files as well.
309
310 External Types
311 ==============
312
313 This is a difficult feature to use but is potentially rewarding in that, once
314 you have generated all the necessary support code by hand, you can include
315 external types in your IDL files with the same status as IDL types.
316
317 To define an external type, use:
318
319 external ip : Ip.ip = "[(<name>,<value>);...]"
320
321 This defines an IDL type called "ip" with OCaml type Ip.ip. The string
322 definition is an assoc list of name-value pairs which define the names of
323 functions to be used as replacement functions for those not generated by the IDL
324 code generator. Note that the type of the OPA-side value is also defined in the
325 assoc list. Here is the current list of names:
326
327 Name Type of value
328 ------------- -------------
329 ocamlstringof <type> -> string
330 ocamlofstring string -> <type>
331 ocamltojson <type> -> JsonTypes.json
332 ocamlfromjson JsonTypes.json -> <type>
333
334 bslwrap <type> -> <opatype>
335 bslunwrap <opatype> -> <type>
336
337 opatype name of the OPA type
338 opastringof <opatype> -> string
339 opaofstring string -> <opatype>
340 opatojson <opatype> -> RPC.Json.json
341 opafromjson RPC.Json.json -> <opatype>
342 opatojsonll <opatype> -> (external) JsonTypes.json
343 opafromjsonll (external) JsonTypes.json -> <opatype>
344
345 Notes
346 -----
347
348 1) The text provided by the <value> string is technically OCaml code but note
349 that you can't currently use complex code here because the string is parsed
461365b [cleanup] Base.String: changed String.split to a much simpler String.sli...
Louis Gesbert authored
350 very simply with String.slice, so you can't have any semicolons or commas
fccc685 Initial open-source release
MLstate authored
351 in your ML code! The above table indicates the types of the resulting
352 OCaml text.
353
354 2) The <opatype> type need in no way correspond to the <type> type. This
355 allows matching of unrelated types (the Ip.ip type is a good example of
356 this).
357
358 3) For an example of the implementation of this code, see the ip.ml and
359 ipopa.opa files in libnet/tests. (Note: don't call your .ml and .opa files
360 by the same name!).
361
362 4) It is unlikely that you will need all of the above values for any
363 particular application. The IDL code generator will inform you if you are
364 missing any of the functions for a particular output code.
365
366 Compiling and running the IDL code generator
367 ============================================
368
369 The IDL code generator executable is called mlidl.native and is installed by
370 build_tools into _build/protocols/mlidl.native.
371
372 The command itself simply takes the name of a single IDL file (the extension
373 should be .mlidl) but note that it may start reading included IDL files. These
374 should already have been processed by the IDL code generator before processing
375 the outer IDL file.
376
377 Compiling the output is somewhat problematical since the dependencies in the
378 generated code are quite complex. Note that you need to include any file
379 dependencies on the command line, particularly, the ML and BSL files for any
380 included types plus have the libraries for any external types. Hopefully, this
381 should be handled by opalang/bld but this is untested.
382
383 Responders
384 ==========
385
386 This is currently the main pay-off for the IDL code generator. From a very
387 simple description file it is possible to generate code which reduces the
388 complex business of managing the transfer of typed data from one network
389 endpoint to another. The IDL annotation:
390
391 val responder_kind : kind -> string
392
393 generates the following values in the OCaml output file:
394
395 val protocol_kind : (kind,string) Hlnet.protocol
396
397 val entrypoint_protocol_kind : (unit option, (kind,string) Hlnet.channel) Hlnet.protocol
398
399 val port_kind : int ref
400 val addr_kind : Unix.inet_addr ref
401 val endpoint_kind : Hlnet.endpoint ref
402 val scheduler_kind : Scheduler.t ref
403
404 val init_responder_kind : int -> Unix.inet_addr -> Scheduler.t -> unit
405
406 These allow the specification of endpoints for the resulting network layer. The
407 port number and network address are actually used to define the endpoint. They
408 are included separately so they can be read back more easily. Use the
409 init_responder_kind function to set all the endpoint values in one go, for
410 example:
411
412 init_responder_kind 12345 Unix.inet_addr_loopback Scheduler.default
413
414 The endpoint and protocol values can then be used by your own communications
415 code but the IDL also generates server and client code which uses these values:
416
417 val respond_server_kind :
418 ('a * 'b) -> (('a * 'b) -> kind -> 'a * 'b * string option * bool) -> unit
419
420 val respond_client_kind :
421 'a -> ('a -> ('a -> kind option * ('a -> string -> ('a -> bool -> unit) -> unit) option -> unit) -> unit)
422 -> ('a -> unit) -> unit
423
424 These are a generic server/client pair and the rather complex types need some
425 explanation. For the server, the actual call to the server looks like:
426
427 respond_server_kind (server_data, connection_data) responder
428
429 This initialises a server. The user-supplied data is divided up into two
430 portions, server_data which persists while the server is active and
431 connection_data which is reinitialised to the value given here at each new
432 connection.
433
434 The "responder" function is a callback function which is called by the server
435 when a value of the given type is received, for example:
436
437 let responder (count,ud_conn) (msg:K.kind) =
438 match msg with
439 | K.Low str -> (count, ud_conn, Some (String.uppercase str), true)
440 | K.Up str -> (count, ud_conn, Some (String.lowercase str), true)
441 | K.Kill -> (sleep 2 @> fun _ -> ()); (count+1, ud_conn, None, count < 2)
442
443 This function is passed the user data (server,connection) plus the type received
444 from the client. It is expected to return a quadruple:
445
446 (<updated server data>,
447 <updated connection data>,
448 <response type>,
449 <continue flag>)
450
451 Most of these are self-explanatory, the response type is the data sent back to
452 the client. In this case the return type was just a string so there is no type
453 conversion done in this case. The continue flag should be "true" if the server
454 is to continue. If "false" then the server terminates.
455
456 The semantics of the client is more complex, the client call looks like:
457
458 respond_client_kind userdata client_cont termination_cont
459
460 Here, the user data is a single value. The client continuation parameter allows
461 control over the connection:
462
463 let client_cont (list_message,_) k =
464 match list_message with
465 (*data*) (*msg*) (*handler*)
466 | [] -> k ([],true) ((Some K.Kill), None) (* Send message and close connection *)
467 | (msg::t) -> k (t,false) ((Some msg), (Some client_handler)) (* Send-receive message *)
468 (*| <whatever> -> k userdata (None, (Some client_handler)) (* Receive message *)*)
469 (*| <whatever> -> k userdata (None, None) (* Close connection immediately *)*)
470
471 This is a continuation function and is passed the user data and the following
472 continuation ("k"). Based on the user data, the client continuation can decide
473 to do various things, the following continuation is passed the updated user data
474 (in this case a list of messages and a status code) plus a pair of an optional
475 query type message for the server and a continuation handler for replies from
476 the server. Based on these values the client code will:
477
478 Message Handler Action
479 ------- ------- ------
480 Some msg Some handler call Hlnet.sendreceive with the message and handler
481 Some msg None call Hlnet.send but does not wait for a reply
482 None Some handler do not send anything but wait with Hlnet.receive
483 None None terminate the connection
484
485 The receive handler has the following semantics:
486
487 let client_handler ((_,last) as userdata) str k =
488 k userdata (not last) (* true=>continue, false=>close connection *)
489
490 It is passed the user data, the message received from the server and a
491 continuation function. It should pass the continuation function the updated
492 user data plus a continuation flag, "false" means terminate the connection.
493
494 Although quite complex, this scheme allows fine control over the server and
495 client by the application, without having to deal with low-level network issues
496 although server/client coordination is still the responsibility of the
497 application code.
498
499 The abstract client model allows the implementation of the following auxiliary
500 client functions:
501
502 val respond_client_single_kind : kind -> (string option -> unit) -> unit
503 val respond_client_send_kind : kind -> (unit -> unit) -> unit
504 val respond_client_receive_kind : (string option -> unit) -> unit
505
506 These are simple one-shot client operations:
507
508 respond_client_single_kind: make a connection to the server, send a single
509 message and handle the reply with the handler function before terminating the
510 connection.
511
512 respond_client_send_kind: connect, send message and close connection.
513
514 respond_client_receive_kind: connect, handle received message and close
515 connection.
516
517 Note that the values received by the handlers are always options. A "None"
518 value means an error in receiving the value.
519
520 Note also that there are significant differences between the BSL implementation
521 of these functions and the OCaml version. This was because I couldn't get OCaml
522 to call the OPA continuation functions without segfaults. The OPA interface is
523 therefore only "pseudo-cps" and works by callback functions instead. This may
524 change if I can ever get OCaml to call an OPA continuation function. See the
525 multi_clients_bsl.opa file for an example.
526
527 Test code and examples
528 ======================
529
530 The standard test script
531 ------------------------
532
533 In libnet/tests there is a script "testmlidl.sh" which allows testing of various
534 features and is intended to be run as a reftester test one day. This has
535 various options for testing specific phases and includes two applications
536 mentioned below.
537
538 Note that this test script attempts to compile using the bare ocaml command line
539 and does not work via opalang/bld. You need a successfully installed
540 opalang/mkinstall for this to work.
541
542 test.mlidl
543 ..........
544
545 This is a bare-bones test which defines one type for each of the types currently
546 handled by the IDL code generator. Note, however, that there are actually a
547 couple of special cases which look like duplicated types but are actually there
548 to test the peculiarities of the OCaml type system (eg. Cons (int * bool) is
549 handled slightly strangely by OCaml). The phase flags are absent from this file
550 to allow the test script to control the phases from the command-line.
551
552 kind.mlidl
553 ..........
554
555 This is a trivial IDL file but it is used in the multi_clients_bsl.opa and
556 multi_clients_idl.ml applications which are IDL implementations of an old Hlnet
557 test program. They use the responder mechanism and illustrate the differences
558 between the two implementations.
559
560 =========
561 N. Scaife
562 26-May-11
563
564
Something went wrong with that request. Please try again.