Skip to content

Commit

Permalink
Unumbrellafication (#1580)
Browse files Browse the repository at this point in the history
* Move src

* Move test/

* Move include/

* Move asn/1

* Move priv/

* Move c_src/

* Move cover.spec

* Move rebar.config.script

* Merge rebar.config

* fixup! Move asn/1

* Rename application resource file

* Fix include directives after moving the sources

* Update .gitignore

* fixup! Merge rebar.config

* fixup! Rename application resource file

* Fix release building and running

* Fix small tests

* Remove remaining occurences of ejabberd application name

* Remove remaining occurences of apps/ejabberd path

* Fix mongoose_cluser module

* Remove remaining references to unumbrellafied parts of the project

* Fix one of ejabberdctl_SUITE cases cleanup

Incorrect cleanup made next test case fail, because of some
offline stanzas pushed.

Document the steps needed to move from umbrella structure
  • Loading branch information
arkgil authored and fenek committed Nov 29, 2017
1 parent 43f59ee commit 33e6392
Show file tree
Hide file tree
Showing 472 changed files with 419 additions and 297 deletions.
30 changes: 15 additions & 15 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -40,21 +40,21 @@ erl_crash.dump
ELDAPv3.asn1db
ELDAPv3.erl
ELDAPv3.hrl
apps/ejabberd/priv/lib/
priv/lib/
*.asn1db
apps/ejabberd/src/XmppAddr.erl
apps/ejabberd/src/XmppAddr.hrl
apps/ejabberd/src/aclocal.m4
apps/ejabberd/src/config.log
apps/ejabberd/src/ejabberd.init
apps/ejabberd/src/ejabberdctl.example
apps/ejabberd/src/eldap_filter_yecc.erl
apps/ejabberd/doc
src/XmppAddr.erl
src/XmppAddr.hrl
src/aclocal.m4
src/config.log
src/ejabberd.init
src/ejabberdctl.example
src/eldap_filter_yecc.erl
doc
apps/*/logs
apps/ejabberd/include/XmppAddr.hrl
apps/ejabberd/include/EJABBERD-MIB.hrl
apps/ejabberd/priv/mibs/EJABBERD-MIB.bin
apps/ejabberd/ebin/ejabberd.app
include/XmppAddr.hrl
include/EJABBERD-MIB.hrl
priv/mibs/EJABBERD-MIB.bin
ebin/ejabberd.app
rel/configure.vars.config
rel/vars.config
compile_commands.json
Expand Down Expand Up @@ -84,7 +84,7 @@ cover_test_preset.log
.mongooseim_combo_dialyzer.plt

## This file is generated by the reload_config tests
apps/ejabberd/test/ejabberd_config_SUITE_data/ejabberd.cfg
test/ejabberd_config_SUITE_data/ejabberd.cfg
fake_*.pem
ebin/
*.coverdata
Expand All @@ -104,7 +104,7 @@ TAGS
*.patch

*.diff
apps/ejabberd/compile_commands.json
compile_commands.json

# ASN.1 rebar3 plugin
asngen
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ For the purpose of this document, we classify changesets/pull requests as:
We strive for Inaka's superb [guidelines](https://github.com/inaka/erlang_guidelines).

MongooseIM has an extensive test suite.
Find **unit** (white box) tests under `/apps/ejabberd/test` and **functional** (black box) tests under `/test/ejabberd_tests/tests`.
Find **unit** (white box) tests under `test/` and **functional** (black box) tests under `test.disabled/ejabberd_tests/tests`.
It's important that tests are comprehensible: consider any preconditions, the test itself, and any postconditions that must hold.
Inspect the existing test suites to see how _we_ make these clear.

Expand Down
7 changes: 3 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,8 @@ SILENCE_COVER = | grep -v "logs.*\\.coverdata"
SILENCE_COVER += | grep -v "Analysis includes data from imported files"
SILENCE_COVER += | grep -v "WARNING: Deleting data for module"
LOG_SILENCE_COVER=$(subst TARGET,$@,TARGET.log 2>&1 || (cat TARGET.log $(SILENCE_COVER); exit 1))
EJABBERD_DIR = apps/ejabberd
XEP_TOOL = tools/xep_tool
EJD_EBIN = $(EJABBERD_DIR)/ebin
EBIN = ebin
DEVNODES = mim1 mim2 mim3 fed1

# Top-level targets aka user interface
Expand All @@ -21,7 +20,7 @@ clean:
-rm rel/vars.config

ct:
@(if [ "$(SUITE)" ]; then ./rebar3 ct --dir apps/ejabberd/test --suite $(SUITE) ;\
@(if [ "$(SUITE)" ]; then ./rebar3 ct --dir test --suite $(SUITE) ;\
else ./rebar3 ct ; fi) > $(LOG_SILENCE_COVER)

rel: certs configure.out rel/vars.config
Expand Down Expand Up @@ -71,7 +70,7 @@ tools/ssl/fake_dh_server.pem:
cd tools/ssl && $(MAKE)

xeplist: escript
escript $(XEP_TOOL)/xep_tool.escript markdown $(EJD_EBIN)
escript $(XEP_TOOL)/xep_tool.escript markdown $(EBIN)

install: configure.out rel
@. ./configure.out && tools/install
Expand Down
27 changes: 0 additions & 27 deletions apps/ejabberd/rebar.config

This file was deleted.

28 changes: 0 additions & 28 deletions apps/ejabberd/rebar.config.script

This file was deleted.

File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
20 changes: 10 additions & 10 deletions doc/Advanced-configuration.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
For advanced configuration use the following files:

* `ejabberd.cfg` for pure MongooseIM settings,
* `ejabberd.cfg` for pure MongooseIM settings,

* `vm.args` to affect the Erlang VM behaviour (performance tuning, node name),
* `vm.args` to affect the Erlang VM behaviour (performance tuning, node name),

* `app.config` to change low-level logging parameters and settings of other Erlang applications.
* `app.config` to change low-level logging parameters and settings of other Erlang applications.

Since you've gotten this far, we assume you're already familiar with Erlang syntax.

Expand Down Expand Up @@ -137,9 +137,9 @@ Retaining the default layout is recommended so that the experienced MongooseIM u
* **Values:** `none`, `tls`
* **Default:** `none`

* **ldap_tls_verify** This option specifies whether to verify LDAP server certificate or not when TLS is enabled.
When `hard` is enabled ejabberd doesn’t proceed if a certificate is invalid.
When `soft` is enabled ejabberd proceeds even if the check fails.
* **ldap_tls_verify** This option specifies whether to verify LDAP server certificate or not when TLS is enabled.
When `hard` is enabled mongooseim doesn’t proceed if a certificate is invalid.
When `soft` is enabled mongooseim proceeds even if the check fails.
`False` means no checks are performed.
* **Values:** `soft`, `hard`, `false`
* **Default:** `false`
Expand Down Expand Up @@ -198,11 +198,11 @@ Retaining the default layout is recommended so that the experienced MongooseIM u
* **Default:** 4096

* [`external` backend options](authentication-backends/External-authentication-module.md#configuration-options)

* [`http` backend options](authentication-backends/HTTP-authentication-module.md#configuration-options)

* [`jwt` backend options](authentication-backends/JWT-authentication-module.md#configuration-options)

* `ldap` backend options are not yet a part of `auth_opt` tuple, so [these parameters](authentication-backends/LDAP-authentication-module.md#configuration-options) are top-level keys in `ejabberd.cfg` file.

* **sasl_mechanisms** (local)
Expand Down Expand Up @@ -441,7 +441,7 @@ Following pool options are recognized - all of them are optional.
* **Regexp format:** Syntax for `_regexp` can be found in [Erlang documentation](http://www.erlang.org/doc/man/re.html) - it's based on AWK syntax. For `_glob` use `sh` regexp syntax.
* **Valid definitions:**
* `all`
* `{user, U}` - check if the username equals `U` and the domain either equals the one specified by the module executing the check or (if the module does a `global` check) is on the served domains list (`hosts` option)
* `{user, U}` - check if the username equals `U` and the domain either equals the one specified by the module executing the check or (if the module does a `global` check) is on the served domains list (`hosts` option)
* `{user, U, S}` - check if the username equals `U` and the domain equals `S`
* `{server, S}` - check if the domain equals `S`
* `{resource, R}` - check if the resource equals `R`
Expand Down Expand Up @@ -476,7 +476,7 @@ Following pool options are recognized - all of them are optional.
### Default language

* **language** (global)
* **Description:** Default language for messages sent by the server to users. You can get a full list of supported codes by executing `cd [MongooseIM root] ; ls apps/ejabberd/priv/*.msg | awk '{split($0,a,"/"); split(a[4],b,"."); print b[1]}'` (`en` is not listed there)
* **Description:** Default language for messages sent by the server to users. You can get a full list of supported codes by executing `cd [MongooseIM root] ; ls priv/*.msg | awk '{split($0,a,"/"); split(a[4],b,"."); print b[1]}'` (`en` is not listed there)
* **Default:** `en`

### Miscellaneous
Expand Down
160 changes: 160 additions & 0 deletions doc/Moving-to-single-app-project-structure.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
# Moving to single application project structure

This document explains how MongooseIM codebase was migrated from the umbrella application structure
to a single application `rebar3` project.

## Moving official MongooseIM to the new structure

### Move all apps/ejabberd subdirectories to the repo root

- mv `apps/ejabberd/src/* src/`
- mv `apps/ejabberd/include/* include`
- mv `apps/ejabberd/priv/* priv`
- mv `apps/ejabberd/test/* test`
- mv `apps/ejabberd/c_src/* c_src`
- mv `apps/ejabberd/asn1/* asn1`

When committing make sure that no new files were added (i.e. they are marked as `renamed`).

### Move tools files

#### cover.spec

`mv apps/ejabberd/src/cover.spec .`

#### rebar.config.script

1. from `apps/ejabberd/src` copy the `MaybeFIPSSupport/1` function
2. paste it into `rebar.config.script`
3. call it on a result of calling `SetupIncludedApps/2` in `rebar.config.script`

#### rebar.config

1. In `rebar.config` remove `{i, "apps"}` line from `erl_opts` tuple
2. Move `{d, xml_nif}` and `{parse_tranform, lager_transform}` lines to `erl_opts` tuple in `rebar.config`
3. From `apps/ejabberd/rebar.config` move all three `erl_first_files`, `xref_checks` and `port_specs` tuples
to `rebar.config`
4. Add `{cover_print_enabled, true}` line below `{conver_enabled, ...` line in `rebar.config`
5. In `rebar.config`, remove the whole `[{override, ejabberd...` structure from `overrides` tuple
6. In `rebar.config`, `pre` tuple, add `{compile, {pc, compile}}` tuple (after `{compile, {asn..`)
7. In `rebar.config`, `post` tuple, add `{clean, {pc, clean}}` tuple (after `{clean, {asn..`)

#### Delete apps/

`rm -rf apps/`

### Changes in source files

#### Application resource files

1. In `src/ejabberd.app.src`
* Change application name to `mongooseim`
* Change `vsn` to `{cmd, "tools/generate_vsn.sh"}`
2. Delete `src/mongooseim.app.src.disabled`
3. `mv src/ejabberd.app.src src/mongooseim.app.src`

### Includes

This is the big one. Since we don't use the umbrella structure anymore, all `-include(..)`
directives pointing to `ejabberd/include` need to be rewritten.

(SED is `sed` on Linuxes or `gsed` on Macs)

* `gsed -i 's/-include_lib("ejabberd\/include\//-include("/' -- src/*.erl src/*.hrl include/*.hrl test/*.erl`
* `gsed -i 's/-include_lib("ejabberd\/src\//-include("/' -- src/*.erl src/*.hrl include/*.hrl test/*.erl`

### Release scripts

In `rel/files/mongooseim` the variable `EJABBERD_SO_PATH` contains an `ejabberd-2.1.8*` fragment.
Replace it with `mongooseim-*`.

In `rel/files/mongooseimctl` the variable `EJABBERD_EBIN_PATH` contains an `ejabberd-*` fragment.
Replace it with `mongooseim-*`.

### Tests & tools

After previous changes, lots of tests fail, mostly because we changed the application name from
`ejabberd` to `mongooseim`. Functions to look for are:
- `application:set_env`
- `application:get_env`
- `application:get_key`
- `application:load`
- `application:start`
- `application:stop`
- `application:ensure_all_started`
- `code:priv_dir`

Those function are also called via RPC from big test suites. A good method to find them all is to
search using `application.*ej` regex.

Files which require changes:
- `include/ejabberd.hrl`
- `src/ejabberd.erl`
- `src/ejabberd_app.erl`
- `src/ejabberd_config.erl`
- `src/mod_version.erl`
- `src/translate.erl`
- `test/ejabberd_config_SUITE.erl`
- `test/ejabberd_helper.erl`
- `test/ejabberd_listener_SUITE.erl`
- `test/zlib_driver_SUITE.erl`
- `test.disabled/ejabberd_tests/README.md`
- `test.disabled/ejabberd_tests/run_common_test.erl`
- `test.disabled/ejabberd_tests/tests/conf_reload_SUITE.erl`
- `test.disabled/ejabberd_tests/tests/connect_SUITE.erl`
- `test.disabled/ejabberd_tests/tests/reload_helper.erl`
- `tools/extract_translations/extract_translations.erl`
- `tools/xep_tool/xep_tool.escript`
- `src/mongoose_cluster.erl` (2 calls to `when_app_stopped/1` function)

We also need to remove remaining occurences of `apps/ejabberd` directory structure. Some of the
files are scripts assembling the path programatically - the trick is to search for the `apps/` keyword.

Affected files:
- `CONTRIBUTING.md`
- `Makefile`
- `elvis.config`
- `src/amp_strategy.erl`
- `src/mod_mam_odbc_arch.erl`
- `test.disabled/ejabberd_tests/Makefile`
- `test.disabled/ejabberd_tests/run_common_test.erl`
- `test.disabled/ejabberd_tests/tests/vcard_SUITE.erl`
- `tools/cd_tools/provision_mysql.sh`
- `tools/cd_tools/provision_pgsql.sh`
- `tools/fill_roster.erl`
- `tools/privacy.erl`
- `tools/travis-setup-db.sh`
- `tools/travis-test.sh`
- `tools/travis-upload-to-s3.sh`
- `tools/zabbix_add_counters.py`

## Moving a MongooseIM's fork to a new structure

This section describes how to move a customized fork of MongooseIM to a new structure.

1. Move all directories from the `apps/ejabberd` directory to the repository root, e.g. `apps/ejabberd/src/`
should be moved to just `src/`. Be sure to move all the important files, because `apps/` directory
will be wiped out completely.

2. Make all the changes described in the `Move tools files` section. If you've made any changes to
`apps/ejabberd/rebar.config` or `apps/ejabberd/rebar.config/script`, be sure to merge these
with `rebar.config` and `rebar.config.script` in the repository root. Remove `apps/` directory.

3. Make all the changes described in the `Application resource files` section.

4. Make all the changes described in the `Includes` section. Make sure that none of the include directives
in your source code (in `src/` and `test/`) has the following structure `-include_lib("ejabberd/include/<header_file>")`.
They should now refer to bare header file names, i.e. `-include(<header_file>)`.

5. Make all the changes described in the `Release scripts` section. Make sure that all customizations
made to release scripts and release process are aware of the new directory structure (i.e. none
of them assumes `apps/ejabberd` path exists).

6. Make all the changes described in the `Tests & tools` section. Since `ejabberd` application was
renamed to `mongooseim`, make sure that all the code which was starting or stopping `ejabberd`,
now does the same for `mongooseim`. You might want to check the following function calls:
- most of the functions from the `application` module
- `code:priv_dir/1`
- calls to `rpc` module which would call functions listed above


Loading

0 comments on commit 33e6392

Please sign in to comment.