Release 1.8.0

Major Changes

Some new projects in the contrib. These are extracts from the libsnapwebsites
library which was getting really big.

Especially, I moved the thread, log, snapcommunicator, some low level headers
which are stand alone and could be useful to all our other libraries.

I also made the cassandra-cpp-driver and libuv1 part of our contribs which
made it possible to move forward to newer versions of Ubuntu (we can compile
our code on 18.04 and 19.04).

Bug Fixes

This version includes a few important bug fixes.

Fast Lists

The list plugin uses a journal and reads a ton of data from the cluster, so it is actually very slow.

We now have an index plugin instead. That new plugin makes use of the index table where we can "manually" create a key which is going to be used to sort entries automatically. The huge difference here is that the sort happens at the time we write that one key to the database. As a result, the lists are instantaneous. The drawback is that we need a special key for each such entry (We also do have special keys for lists, though) and that key can be slow to calculate. However, seeing how slow the management of list items is, this new method will look extremely fast in comparison. Once the Save is done, your lists were updated.

The one potential problem is that one item could be part of many lists and that could be a problem as in we are going to add the item to all those lists one after the other which the client is waiting for an answer...

Background Services

Started work on having services create a PID file. This is useful for systemd which uses the PID file to know that a service is running and to save the identifier (pid) of the process to kill when asked to STOP a service.

libaddr Buffer Overflow

The libaddr had a case where the length of a buffer was not being checked.

libcasswrapper Multi-thread Bug

The libcasswrapper project would not properly handle switching between various queries, especially in a multithreaded environment. The fact is that one variable was not correctly projected. I changed a few things and actually could make one variable thread specific (class specific, really, so there is still a need for protection, but it's very light).

Note that only cassview and some other command line tools were affected. Snap! itself is not using that feature so it had no problem in this area.

users Plugin

The users plugin is responsible for handling login data. One of the tests in there would check the "LOCALES" variable when it was supposed to checked "VERIFIED_ON". This fixes various display feature which would think that the user was not yet verified.

Note that the login feature, passwords, etc. were not affected (it was not a security issue).

Snap Manager

The snapmanager had several display bugs and problems with reload that were fixed.

We started work on a new project (snaprfs) which will also help us in building version 2 which will be a lot more AJAX and much lighter for the interface (i.e. load one screen instead of all of them). It will also be capable of handling work in a batch instead of the current method which is immediate.

I also fixed a couple of refresh bugs: problems with variables that would not call the refresh functions.

The host name may not be defined, in which case snapmanagerdaemon would crash.

Fixed the Cassandra plugin which forced us to define an IP address. If not present, it would crash preventing us from adding the address in the first place.

snapstop Missing PID

When the snapstop command is called from systemctl, it may not have a MAINPID variable. When that's the case, snapstop gets called with snapstop --service on its own. This failed in older versions. Now we ignore the option if the parameter is not specified.

snaplock Supported Messages

The snaplock binary did not properly report that it supported the CLUSTERUP and CLUSTERDOWN messages.

"Package[s]" in snapwatchdog

The default list of snapwatchdog plugins included the Packages plugin spelled "package", without the "s".

snapcommunicator Cluster File Bug

The communicator would not save the cluster status until it received its first CLUSTERSTATUS message. This means the snapmanagerdaemon could not properly report the cluster status.

Links Bug

In some cases, the deletion of a link would not be complete. There was a return in a function which should have been a continue to make sure all possible cases were properly handled.

Message Box

Whenever too many messages were shown, they would disappear since the popup has a limited size.

Content Plugin

A test in content::modified_content_impl() was inverted (i.e. return true instead of false and vice versa).

I inverted the test so it is now as it was expected.

Reproducible Binaries

I removed the __DATE__ and __TIME__ parameters. Instead, I use our UTC_BUILD_DATE and UTC_BUILD_TIME which are now directly available in our snapcmakemodules.

This allows for reproducible binaries. If you know when something was compiled, then you can recompile and get the same date and time information.

Our macros also include one variable per field (year, month, day, hour, minute, seconds) which gives us a way to create a copyright notice which changes automatically.

Project: snapdev

This small project is a set of C++ header files that implement various
algorithms and functions that we use throughout our projects.

This library is already 18 headers.

Next step: We are slowly removing duplicates from the contribs (such as
the not_reached.h).

Project: cppthread

Our the C++ thread implementation from libsnapwebsites is now a project
on its own.

It includes the functions to handle processes with pipes to
stdin/stdout/stderr.

Next step: completely remove the snap_thread.cpp/h from the Snap!
project.

Project: eventdispatcher

Our snap_communicator.cpp/h was enormous and we wanted to break it up.
This is now the eventdispatcher project.

Broken up in separate files, it is 80 files (not counting the CMakeLists.txt).
The project includes its own version and utilities.

It includes all the features we had available in the libsnapwebsites.
We will be adding more functionality to track changes to directories. For
example, you will be able to get an event whenever one of your configuration
file may have changed. That way you can automatically reload those files
and be up to date.

Next step: completely remove the snap_communicator.cpp/h from the Snap!
project. I first need to switch to snaplogger and cppthread, though.

Project: snaplogger

Our replacement to the libsnapwebsites log.cpp/h without the need for the
log4cplus v1.2. The log4cplus project has not evolved much since 1.2 came
out and the properties do not offer us a way to make global changes easily.

This new project, snaplogger, makes use of advgetopt which is much more
versatile. This also means we are walking away from the Java properties
to move to a more Unix like configuration file.

This version already includes a thread to support asynchrounous logging
and we have plans to have a 1 to 1 compatible log server which is going
to be implemented using the eventdispatcher library.

All the useful features available in log4cplus were replicated and many
more added. Especially, the format support for the log messages is
very easily extensible so other projects can add support for their own
parameters.

Note: the snapwebsites projects have not yet been converted, however, I
already make use of the snaplogger in the eventdispatcher and it works
just fine. The cppthread uses the advgetopt log functions which allow
for a callback to be setup. The snaplogger sets that callback so it
will captures those logs automatically for us. In other words, even
though snaplogger depends on advgetopt and cppthread, it is capable of
logging messages from those two lower level libraries as if they knew
about snaplogger.

Next step: completely remove the log.cpp/h from the Snap!
project. I first need to switch to cppthread, though.

Project: libutf8

I finally made a copy of my utf8 code from wpkg to a library in Snap!.

I enhanced the project (of course...) and make use of it in many places.
This is a replacement to the QString which we've been using so far.

One of the most interesting feature is the bidirectional iterator which
allows us to walk a UTF-8 string and get char32_t characters. This is
going to be extermely useful in many places where we need to know of
the Unicode character. We already use that feature in several places
such as advgetopt.

Project: snapcatch2

The old catch in 16.04 was... well... old and it was not compatible with
the version of catch available on 18.04 and newer versions of Ubuntu. This
meant we could not really compile our tests on those platforms.

Now we have our own snapcatch2 project which is a copy of a specific version
of catch2 which is going to work with all of our projects.

We also include a snapcatch2.hpp header which gives us the ability to
create a main() in a jiffy. It also gives all our tests a similar set
of options such as the --seed <seed> command line option.

Project: advgetopt

This project was very heavily updated. The main goal was to add the
configuration load and save features we had in our libsnapwebsites.

As a result, we now have a very large number of new features such as
the possibility to override parameters using filenames in a ".d" sub-folder
using files names that start with two digits (##-.conf). This is
a huge advantage since it will give our projects a way to change settings
of another projects just by installing a file under /etc.

Another major addition, the system offers ways to add options dynamically.
This is now used to add system options (such as --help and --version)
and in snaplogger, to add log specific command line options (i.e. the
--debug automatically reduces the log level to DEBUG.)

Project:: libcasswrapper

There has been a bug in the cassview GUI tool where trying to load
different rows too quickly would crash the software.

This was due to an incorrect access to a non-protected variable in our
multi-thread version of the library. This is now fixed.

I also applied a few fixes where some parameters may be null or undefined
in some way and thus they should not be used. There are still more functions
where such tests could be added.

Project: libexcept

I found a post on Stackoverflow which had a code sample on how to demangle
C++ function names without the need to run c++filt. This just requires a
few system calls. I added a demangle.cpp/h set of files to handle that
work. We now get stack traces which are much faster with demangled names.

Changelogs (snapwebsites only)

• Enabled the snap-apache2-extra-options (use a semaphore as locks).
• Added default ServerName (http://127.0.0.1:80) to avoid the Apache warning.
• Changed the snapmanager-dpkg-update hook to ignore all errors.
• Fixed the users token VERIFIED ON would improperly use LOCALES.
• Added support for users::username.
• Fixed the gathering of the signal_secret parameter in the self plugin.
• Updated a couple of files to work with the new addr exceptions.
• Made snaplogger a dependency of snapbase (to have shredlog).
• Use shredlog to delete secure logs and the snapmanager.conf files.
• Comment out the shred option in the logrotate, with SSD it's useless.
• Removed warning about non-locked log file, it is locked now.
• Fixed the locale so it finds the libraries (i.e. correct namespace).
• Changed the TIME and DATE with our UTC_BUILD_TIME/DATE vars.
• Fixed the end of the server option list.
• Added libutf8 as a new dependency.
• Removed the headers I moved to snapdev.
• Clean up some comments that were just old code.
• Removed a few fprintf() we still had left in a couple of tests.
• Applied a fix to get the ICU to work on 19.04.
• Fixed various OpenSSL calls (deprecated, ignored parameters, etc.)
• Added #include in various files.
• Made many fallthrough C++17 compatible.
• It looks like g++ 8.x wants explicit noexcept.
• The ICU library requires a namespace in g++ 8.x.
• Removed the YAML_CPP_INCLUDE_DIR because for now it's wrong on 19.04.
• Added snapdev dependency for build to work.
• Updated the command line tools to compile against the new advgetopt.
• Added in=C++ to the MAPPING_EXTENSION to all doxy files.
• Added the PROJECT_BRIEF description in some doxy files.
• Updated all doxy files to 1.8.11.
• Added options to the debian/source to avoid including tmp & .git in the source package.
• Make use of the UTC_BUILD_YEAR for our copyright notices.
• Added a README.md for the library mb tests.
• Added a test to verify whether the snap_thread FIFO should be implemented with a vector or a queue. The queue wins big time.
• Added an enumeration function to enumarate the children of a page and call a callback function. The function ensures that the pages are NORMAL.
• When removing the list::type link from a page, the list plugin automatically deletes all the list related fields, including the ordered list.
• Changed a return in a continue in the links::delete_link() function.
• Added support for a pool of threads and workers.
• SNAP-684: Added the snapclusterstatus command line tool.
• SNAP-684: Included a man.1 page for snapclusterstatus.
• SNAP-684: Made snapcommunicator save the snap-cluster.txt file with its most current status.
• SNAP-684: Fixed a bug in a one computer environment the cluster status was not set until we receive a very first CLUSTERSTATUS message.
• EX-204: Fixed the path used to check the field, was using wrong variable.
• EX-204: Made the upload reset button appear only if a flag is turned on.
• In the snapmanagercgi output, fixed the page title so it includes the host name.
• Fixed the double "(Select Host)" message from the status.
• SNAP-181: Applied a fix as the whitelist would not update automatically, we have to save the value in the config (because it's static memory.)
• SNAP-181: Fixed the signal_secret which would also not get updated.
• SNAP-350: Fixed list of supported message in snaplock, the CLUSTERUP and CLUSTERDOWN had to be included.
• SNAP-478: Fixed the type for result for a DELETE in the index plugin.
• SNAP-478: Fixed the index case, when indexing the key may change, to avoid duplicates, we must delete existing entries that do not match the new 'column1' value.
• SNAP-110: Fixed the menu, do not show Host Watchdog when no host is selected.
• SNAP-181: added a field to edit the "whitelist" variable from iplock.
• SNAP-526: added a field in snapmanager to edit the signal_secret field.
• SNAP-478: added the index plugin to the packages.
• SNAP-413 and SNAP-421 and SNAP-454: Moved the snapcreatecontext functionality to snapdbproxy.
• SNAP-413: Moved the snapcreatetables functionality to snapdbproxy.
• SNAP-413: Fixed the NEWTABLE implementation.
• SNAP-267: Added an XSD to verify tables.xml files and updated the CMakeLists.txt.
• SNAP-478: implemented the index plugin to have instant "lists".
• SNAP-478: added a link() function in snap_expr, useful for indexes.
• SNAP-679: added a "temporary" fix to the fail2ban nice value.
• SNAP-492: implemented the support for a PID file for snapserver, snabackend, snacommunicator, and snapwatchdog (so far.)
• SNAP-641: Started work on a fix to not leak messages to the wrong user.
• SNAP-526: added the ability to add a secret code in a UDP message to make them a little more secure.
• Changed the sendmail algorithm to allow for ignoring the fact that the user's email address failed with a bounce when the user is an admin.
• When marking a session as used up, save the date when the event happened.
• Install the unblock-ip script even though it's not very useful as it stands.
• Converted the snapcommunicator address test to make use of addr::addr (the domain name includes the port!)
• Added a script one can use to search for users who have a bounced email.
• Add the address to the addrinfo_t unique_ptr<>() because it could be that we had an error (r != 0) and a pointer (addrinfo != null).
• Verify buffer length in get_client_addr().
• Added support for keepalive in the client BIO.
• Properly initialize a variable in case BIO_get_fd() fails.
• Applied a fix to the snapmanager so it does not crash if the name of the host specified on the query string is unknown.
• Fixed the cassandra plugin, if the "listen_address" is not defined it would crash when trying to get the parameter when it's not defined.
• Applied a fix to the snapstop making the --service parameter optional. (The --service "$MAINPID" becomes --service when $MAINPID is empty.)
• Added /etc/vim/vimrc.local so we get our search, line numbers, etc.
• Made the attach/detach session RAII so it at least happens on exceptions.
• Added function to create a complete page in one call. (editor create_page)
• Fixed the f_cassandra_connect_timer_index calculation.
• Made the type found by find_type_with() available through another call.
• Fixed the default watchdog plugins list, "package" was wrong, "s" missing.
• Applied a fix to the message box, it adds a scrollbar when many messages appear in the list.
• Fixed HUGE bug in content::modified_content_impl() where I would return false if the snap_child object was ready. That test was inverted!
• Various clean ups.
• Remove reference to /run/snapwebsites from snapbuild.service.
• Make use of snapbase.tmpfile to create the /run/snapwebsites folder.
• Prevent throwing when a plugin version is not correct.