XMPP codec should be modular #9

zinid · 2017-02-07T11:07:35Z

XMPP codec should allow third-party developers to create their own submodules and plug them into xmpp_codec.
This is implemented in 3976a85
Here is a brief description for developers about how they could use it:

Clone xmpp library somewhere:
$ git clone https://github.com/processone/xmpp
Compile it:
$ make
Add new elements to specs/xmpp_codec.spec with proper module attribute set
Recompile the codec:
$ make spec
The generator will create new file(s) in src directory. Those files are your submodules. Grab them to the working directory of your project.
The generator also will generate new records (with specs) inside include/xmpp_codec.hrl. Grab those too in your project (either put them in your *.hrl files or in your *.erl files directly).
In runtime, call to xmpp:register_codec/1 in order to register your submodule within xmpp_codec.

Done!
Not very clear, eh? OK, here is an example.

Let's say we want to add submodule for the following element:

<foo x='1' xmlns='ns:foo'/>

Clone and compile the repo, open specs/xmpp_codec.spec and put the following spec somewhere inside it (the order doesn't matter):

-xml(foo,
     #elem{name = <<"foo">>,
           xmlns = <<"ns:foo">>,
           module = foo,
           result = {foo, '$x'},
           attrs = [#attr{name = <<"x">>}]}).

Now type make spec in order to recompile the specification. New file will be created inside src directory:

$ git status
...
Untracked files:
   ...
   src/foo.erl

Copy this foo.erl into your working directory.
Now, consult the recompiled xmpp_codec.hrl:

$ git diff include/xmpp_codec.hrl
...
+-record(foo, {x = <<>> :: binary()}).
+-type foo() :: #foo{}.
+
...

Put this record and type definition somewhere in your files.
Now, you need to register your submodule (foo) during startup. This should look something like this:

-module(mod_foo).
...
start(Blah, ...) ->
    ...
    xmpp:register_codec(foo),
    ...

Note, that you need to call xmpp:register_codec/1 again, if you have your submodule (foo) recompiled and reloaded in runtime.
Also, you may want to unregister the submodule during shutdown procedure, although, strictly speaking, this is not required:

stop(...) ->
    xmpp:unregister_codec(foo)

Now you have everything done to use your new "subcodec".
Have fun ;)

The text was updated successfully, but these errors were encountered:

grizzly-monkey · 2017-02-08T06:49:19Z

Great ... thanks will try it out soon

grizzly-monkey · 2017-02-08T06:51:22Z

tag in rebar is
{deps, [{fast_xml, ".*", {git, "https://github.com/processone/fast_xml", "23f60e2942113fcf5d87531c081ef47329c343ef"}

should it be some version tag ? @zinid

zinid · 2017-02-08T07:03:33Z

No, we tag it during release, later.

loguntsov · 2017-05-16T18:18:10Z

This is example of script to build custom XMPP codec module for this library:

#!/bin/bash
rm -rf tmp
mkdir tmp
cd tmp
git clone https://github.com/processone/xmpp.git &&
cd xmpp &&
git checkout 1.1.9 &&
cat ../../specs/sec_history_xmpp.spec >> specs/xmpp_codec.spec &&
cp include/xmpp_codec.hrl include/xmpp_code_old.hrl &&
make all &&
make spec &&
cp src/sec_history_xmpp.erl ../../src/sec_history_xmpp.erl &&
echo -e "%% This file was generated automatically by compile_xmpp_spec.sh script\n\n" > ../../include/sec_xmpp_codec.hrl &&
diff -n include/xmpp_code_old.hrl include/xmpp_codec.hrl | grep -v "^\a" | grep -v "() |" >> ../../include/sec_xmpp_codec.hrl &&
cd ../..
rm -rf tmp

The project structure:

├── include
│   └── sec_xmpp_codec.hrl
├── Makefile
├── script
│   └── compile_xmpp_specs.sh
├── specs
│   └── sec_history_xmpp.spec
└── src
    └── sec_history_xmpp.erl

spec folder should contains specification additional tags.
script/compile_xmpp_specs.sh --- the script described above.
Script will generate 2 files:
./src/sec_history_xmpp -- source code for codec.
./include/sec_xmpp_codec.hrl --- header file with description of records and types.

Makefile can be like this:

.PHONY: spec
spec:
	script/compile_xmpp_specs.sh

So just run:
make spec
to build your own specification for xmpp.
After that you can use this module as:
xmpp:register_codec(sec_history_xmpp.erl)

loguntsov · 2017-05-16T18:20:38Z

By the way. All these flow are big crutches. XMPP library MUST allow build CUSTOM MODULES for all developers.

zinid · 2017-05-16T18:35:07Z

MORE CAPS

ltAldoRaine · 2018-02-05T11:30:59Z

how to make custom iq spec? for example i have custom iq handler with query ns "NS-CUSTOM-MODULE"

zinid · 2018-02-05T20:11:51Z

@ltAldoRaine you can show an example of your IQ payload, so I can write some spec you can start with.

ltAldoRaine · 2018-02-05T20:21:44Z

<iq type=“get”><query xmlns=“jabber:iq:conversation”><start>0</start><limit>10</limit></query></iq>

I did everything as you described. after restarting ejabberd there was error which was throwed by xmpp:register_codec()

7:47:40.261 [critical] Problem starting the module mod_conversation for host ****
 options: []
 error: undef
[{conversation,module_info,[md5],[]},
 {xmpp_codec,register_module,2,[{file,"src/xmpp_codec.erl"},{line,121}]},
 {mod_conversation,start,2,[{file,"src/mod_conversation.erl"},{line,42}]},
 {gen_mod,start_module,4,[{file,"src/gen_mod.erl"},{line,200}]},
 {lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
 {gen_mod,start_link,0,[{file,"src/gen_mod.erl"},{line,79}]},
 {supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,365}]},
 {supervisor,start_children,3,[{file,"supervisor.erl"},{line,348}]}]
17:47:40.261 [critical] ejabberd initialization was aborted because a module start failed.
Problem starting the module mod_conversation for host ****
 options: []
 error: undef
[{conversation,module_info,[md5],[]},
 {xmpp_codec,register_module,2,[{file,"src/xmpp_codec.erl"},{line,121

Crash dump is being written to: /usr/local/var/log/ejabberd/erl_crash_20180205-174736.dump...done
[os_mon] memory supervisor port (memsup): Erlang has closed

zinid · 2018-02-06T12:03:34Z

What Erlang version are you using?
Did you compile your spec so you get conversations.erl as an output?
Did you compile conversations.erl? Does Erlang see conversations.beam (use code:which/1 to check)?
Post your spec somewhere so I can check its correctness.

ltAldoRaine · 2018-02-06T12:18:31Z

Erlang/OTP 20 [erts-9.2]
yes it was generated in cloned xmpp src folder so i moved it to ejabberd deps/xmpp/src folder
i think i missed that part.. where should i compile it ? in erlang ebin folder or ejabberd ebin folder

zinid · 2018-02-06T12:23:33Z

You should compile it inside your project. Then you just set CONTRIB_MODULES_CONF_DIR option in ejabberdctl.cfg which points to the beam files of your project.

zinid · 2018-02-06T12:25:00Z

You can of course compile it within ejabberd project (just put your ERL file inside src directory and run make).

ltAldoRaine · 2018-02-06T12:26:39Z

Thanks ! I will try second option..my modules(not xmpp-codec) is also in ejabberd src folder

ltAldoRaine · 2018-02-06T17:16:57Z

It worked ! :) now ejabberd is receiving my iq stanza.

rasel800 · 2019-01-31T15:09:55Z

I was experimenting with this example and still can't make it work.
Using ejabberd-18.12 in Mac, used OS X package to install ejabberd.

So, the steps performed:

Clone XMPP and make
Added spec below in specs/xmpp_codec.spec and make spec

-xml(foo,
     #elem{name = <<"foo">>,
           xmlns = <<"ns:foo">>,
           module = foo,
           result = {foo, '$x'},
           attrs = [#attr{name = <<"x">>}]}).

Moved src/foo.erl to working directory where mod_foo.erl resides. foo.erl below:

%% Created automatically by XML generator (fxml_gen.erl)
%% Source: xmpp_codec.spec

-module(foo).

-compile(export_all).

do_decode(<<"foo">>, <<"ns:foo">>, El, Opts) ->
    decode_foo(<<"ns:foo">>, Opts, El);
do_decode(Name, <<>>, _, _) ->
    erlang:error({xmpp_codec, {missing_tag_xmlns, Name}});
do_decode(Name, XMLNS, _, _) ->
    erlang:error({xmpp_codec, {unknown_tag, Name, XMLNS}}).

tags() -> [{<<"foo">>, <<"ns:foo">>}].

do_encode({foo, _} = Foo, TopXMLNS) ->
    encode_foo(Foo, TopXMLNS).

do_get_name({foo, _}) -> <<"foo">>.

do_get_ns({foo, _}) -> <<"ns:foo">>.

pp(foo, 1) -> [x];
pp(_, _) -> no.

records() -> [{foo, 1}].

decode_foo(__TopXMLNS, __Opts,
	   {xmlel, <<"foo">>, _attrs, _els}) ->
    X = decode_foo_attrs(__TopXMLNS, _attrs, undefined),
    {foo, X}.

decode_foo_attrs(__TopXMLNS, [{<<"x">>, _val} | _attrs],
		 _X) ->
    decode_foo_attrs(__TopXMLNS, _attrs, _val);
decode_foo_attrs(__TopXMLNS, [_ | _attrs], X) ->
    decode_foo_attrs(__TopXMLNS, _attrs, X);
decode_foo_attrs(__TopXMLNS, [], X) ->
    decode_foo_attr_x(__TopXMLNS, X).

encode_foo({foo, X}, __TopXMLNS) ->
    __NewTopXMLNS =
	xmpp_codec:choose_top_xmlns(<<"ns:foo">>, [],
				    __TopXMLNS),
    _els = [],
    _attrs = encode_foo_attr_x(X,
			       xmpp_codec:enc_xmlns_attrs(__NewTopXMLNS,
							  __TopXMLNS)),
    {xmlel, <<"foo">>, _attrs, _els}.

decode_foo_attr_x(__TopXMLNS, undefined) -> <<>>;
decode_foo_attr_x(__TopXMLNS, _val) -> _val.

encode_foo_attr_x(<<>>, _acc) -> _acc;
encode_foo_attr_x(_val, _acc) ->
    [{<<"x">>, _val} | _acc].

Compared the include/xmpp_codec.hrl before vs. after and added the difference in mod_foo.erl

-record(foo, {x = <<>> :: binary()}).
-type foo() :: #foo{}.

So mod_foo.erl is like below (just for output IQ to log):

-module(mod_foo).
-include("logger.hrl").
-behaviour(gen_mod).
-export([start/2, stop/1, process_local_iq/1]).
-record(foo, {x = <<>> :: binary()}).
-type foo() :: #foo{}.
start(Host, Opts) ->
    xmpp:register_codec(foo),
    gen_iq_handler:add_iq_handler(ejabberd_local, Host, <<"ns:foo">>, ?MODULE, process_local_iq),
    ok.
stop(Host) ->
    xmpp:unregister_codec(foo),
    gen_iq_handler:remove_iq_handler(ejabberd_local, Host, <<"ns:foo">>),
    ok.
process_local_iq(IQ) ->
    ?INFO_MSG("IQ received:~p", [IQ]),
    ignore.

Compiled and loaded mod_foo module by ejabberdctl module_install mod_foo, with some warnings that I've ignored.

~/.ejabberd-modules/sources/mod_foo/src/foo.erl:5: Warning: export_all flag enabled - all functions will be exported
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:3: Warning: undefined callback function depends/2 (behaviour 'gen_mod')
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:3: Warning: undefined callback function mod_options/1 (behaviour 'gen_mod')
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:7: Warning: type foo() is unused
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:9: Warning: variable 'Opts' is unused

Sending XML:

<iq type="set" id="1234" to="localhost">
    <foo x='1' xmlns='ns:foo'/>
</iq>

Receiving XML:

<error code="503" type="cancel">
        <service-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas" />
        <text xmlns="urn:ietf:params:xml:ns:xmpp-stanzas" lang="en">No module is handling this query</text>
</error>

What am I missing here?
Do I need to replace the xmpp_codec.hrl and xmpp_codec.beam with the new compiled ones?

Would appreciate any suggestions please.

zinid · 2019-01-31T16:31:56Z

What am I missing here?

The module foo.erl is generated automatically by the xmpp_codec, you can find it inside src directory of xmpp sources. You MUST NOT modify it. And your own module should have a different name, obviously, like mod_foo.erl.

rasel800 · 2019-02-01T04:29:55Z

@zinid, truly appreciate the prompt response.
I have updated the steps above with as much details as possible according to your suggestion and still can't make it work.

I wish there were an easier way to do this and proper documentation of the steps with examples.
Would appreciate any suggestion please.

zinid · 2019-02-01T07:43:52Z

Compiled and loaded

What do you mean by "loaded"? Did you put it into ejabberd.yml?

rasel800 · 2019-02-01T09:00:53Z

Compiled and loaded

What do you mean by "loaded"? Did you put it into ejabberd.yml?

No, I meant compiled and installed the module mod_foo using ejabberdctl module_install mod_foo command.
https://docs.ejabberd.im/admin/guide/managing/#ejabberdctl-commands

Actually that was the problem, updated ejabberd.yml and restarted server, working fine now!

Wondering why the ejabberdctl module_install mod_foo didn't get the module started in the first place?
Although I see ejabberdctl module_upgrade mod_foo is working fine for runtime compilation and reloading the module after adding mod_foo in ejabberd.yml and restarting server once.

Anyway, thank you very much @zinid, truly appreciate.

zinid · 2019-02-01T11:45:55Z

Wondering why the ejabberdctl module_install mod_foo didn't get the module started in the first place?

Because a module typically requires some start options.

Zedonboy · 2019-08-24T17:57:15Z

If you are still having this issue, check.. here

zinid added the enhancement label Feb 7, 2017

zinid self-assigned this Feb 7, 2017

zinid closed this as completed Feb 7, 2017

zinid mentioned this issue Feb 7, 2017

How to add custom stanza #6

Closed

zinid mentioned this issue May 5, 2017

My custom attributes are being eliminated from messages since ejabberd 16.12 processone/ejabberd#1692

Closed

zinid mentioned this issue Jun 14, 2017

Ejabberd - Create a custom stanza (GET/SET Subtags) processone/ejabberd#1779

Closed

loguntsov mentioned this issue Jun 21, 2017

Don't compiles spec for xmpp #13

Closed

zinid mentioned this issue Jul 20, 2017

Adding Custom IQ Handler not working processone/ejabberd#1877

Closed

ltAldoRaine mentioned this issue Feb 6, 2018

offline_message_hook crashed when running my custom module processone/ejabberd#2269

Closed

zinid mentioned this issue Apr 11, 2018

Issue modifying subtags #27

Closed

zinid mentioned this issue Jul 5, 2018

Custom IQ Handler not working processone/ejabberd#2508

Closed

zinid mentioned this issue Oct 2, 2018

Improve codec generation from third-party specs #36

Open

relbraun mentioned this issue Jun 22, 2020

Can't handle a custom IQ stanza processone/ejabberd#3306

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

XMPP codec should be modular #9

XMPP codec should be modular #9

zinid commented Feb 7, 2017 •

edited

Loading

grizzly-monkey commented Feb 8, 2017

grizzly-monkey commented Feb 8, 2017

zinid commented Feb 8, 2017

loguntsov commented May 16, 2017 •

edited

Loading

loguntsov commented May 16, 2017

zinid commented May 16, 2017

ltAldoRaine commented Feb 5, 2018

zinid commented Feb 5, 2018

ltAldoRaine commented Feb 5, 2018 •

edited by mremond

Loading

zinid commented Feb 6, 2018 •

edited

Loading

ltAldoRaine commented Feb 6, 2018 •

edited

Loading

zinid commented Feb 6, 2018

zinid commented Feb 6, 2018

ltAldoRaine commented Feb 6, 2018 •

edited

Loading

ltAldoRaine commented Feb 6, 2018 •

edited

Loading

rasel800 commented Jan 31, 2019 •

edited

Loading

zinid commented Jan 31, 2019

rasel800 commented Feb 1, 2019

zinid commented Feb 1, 2019

rasel800 commented Feb 1, 2019 •

edited

Loading

zinid commented Feb 1, 2019 •

edited by mremond

Loading

Zedonboy commented Aug 24, 2019

XMPP codec should be modular #9

XMPP codec should be modular #9

Comments

zinid commented Feb 7, 2017 • edited Loading

grizzly-monkey commented Feb 8, 2017

grizzly-monkey commented Feb 8, 2017

zinid commented Feb 8, 2017

loguntsov commented May 16, 2017 • edited Loading

loguntsov commented May 16, 2017

zinid commented May 16, 2017

ltAldoRaine commented Feb 5, 2018

zinid commented Feb 5, 2018

ltAldoRaine commented Feb 5, 2018 • edited by mremond Loading

zinid commented Feb 6, 2018 • edited Loading

ltAldoRaine commented Feb 6, 2018 • edited Loading

zinid commented Feb 6, 2018

zinid commented Feb 6, 2018

ltAldoRaine commented Feb 6, 2018 • edited Loading

ltAldoRaine commented Feb 6, 2018 • edited Loading

rasel800 commented Jan 31, 2019 • edited Loading

zinid commented Jan 31, 2019

rasel800 commented Feb 1, 2019

zinid commented Feb 1, 2019

rasel800 commented Feb 1, 2019 • edited Loading

zinid commented Feb 1, 2019 • edited by mremond Loading

Zedonboy commented Aug 24, 2019

zinid commented Feb 7, 2017 •

edited

Loading

loguntsov commented May 16, 2017 •

edited

Loading

ltAldoRaine commented Feb 5, 2018 •

edited by mremond

Loading

zinid commented Feb 6, 2018 •

edited

Loading

ltAldoRaine commented Feb 6, 2018 •

edited

Loading

ltAldoRaine commented Feb 6, 2018 •

edited

Loading

ltAldoRaine commented Feb 6, 2018 •

edited

Loading

rasel800 commented Jan 31, 2019 •

edited

Loading

rasel800 commented Feb 1, 2019 •

edited

Loading

zinid commented Feb 1, 2019 •

edited by mremond

Loading