Merge branch 'beta' into production

* beta:
  Update translations from transifex.
  Update POT files using the production database
  Install pkg-config to probe ICU headers/libraries (#1360)
  Fix MBS-10572: Have one Jed bundle per language (#1359)
  MBS-10571: Fix "localized" ModBot email display (#1358)
  Eslint fixes: no-prototype-builtins
  Document musicbrainz_selenium creation
  MBS-10567: Limit series types as per DB constraint (#1351)
  MBS-10560: Show full country names in release edits (#1353)
  MBS-10536: Remove span.name-variation around "see all" releases link (#1354)
  Remove unneeded extra newlines
  MBS-10365: Convert ReleaseList to react-table
  Update POT files
  Update translations from Transifex
  MBS-10565: Fix item_entity_type field name (#1350)
  Bump Flow to 0.116.0
  MBS-10553: Translate report reason only if needed (#1346)
  Handle when maps service access token is not set
  Document MAPBOX_ACCESS_TOKEN in sample config
  Rename merge_entities to transfer_to_merge_target
  MBS-10522: Save subscriptions to merge destination
  MBS-10501: Show collaborator avatars on collection (#1342)
  MBS-9491: List genre on tag search results (#1064)
  More documentation tweaks. - Changed some of the wording around the PostgreSQL user. - Avoid using raw URLs. - Link to mb.org instead of ftp.mb.org for obtaining database dumps.   - Unfortunately, that page does not include any info about the sample data set,     so that section needs to describe how to adjust the download URL. - Dropped information about a CPAN bug that was resolved in 2013.
  Eslint sort-keys fixes
  MBS-10552: Add Deezer links to the sidebar
  MBS-10492: Display leading 0 in minute and second (#1341)
  Reworked the translation section. The prerequisites now list detailed Transifex information (and make it clear it is NOT required to use the translations that are under version control). Added a sentence explaining the format for languages.
  Adjusted database user documentation. Instead of describing how to change a user's password, simply mention that the database creation script will also create the database user if needed.
  Bump Flow to 0.115.0
  Updated the installation guide.

HACKING.md

@@ -187,6 +187,12 @@ These are a bit more involved to set up:
 
  * Run ./script/create_test_db.sh and ./script/compile_resources.sh again.
 
+ * Create the `musicbrainz_selenium` database using `musicbrainz_test` as
+   a template:
+
+       $ # On Ubuntu, use sudo -u postgres
+       $ createdb -O musicbrainz -T musicbrainz_test -U postgres musicbrainz_selenium
+
 With the above prerequisites out of the way, the tests can be run from the
 command line like so:
 

INSTALL.md

@@ -172,7 +172,8 @@ Below outlines how to setup MusicBrainz server with local::lib.
             libpq-dev \
             libxml2 \
             libxml2-dev \
-            cpanminus
+            cpanminus \
+            pkg-config
 
 2.  Enable local::lib
 
@@ -228,7 +229,7 @@ Creating the database
     To build our collate extension you will need libicu and its development
     headers, to install these run:
 
-        sudo apt-get install libicu-dev
+        sudo apt-get install libicu-dev pkg-config
 
     With libicu installed, you can build and install the collate extension by
     running:
@@ -251,8 +252,9 @@ Creating the database
     users (whoever your web server/crontabs run as), to one database (the
     MusicBrainz Database), as one PostgreSQL user. The PostgreSQL database name
     and user name are given in DBDefs.pm (look for the `READWRITE` key).  For
-    example, if you run your web server and crontabs as "www-user", the
-    following configuration recipe may prove useful:
+    example, if you run your web server and crontabs as "www-user", and you have
+    kept the default PostgreSQL user name ("musicbrainz"), the following
+    configuration recipe may prove useful:
 
         # in pg_hba.conf (Note: The order of lines is important!):
         local    musicbrainz_db    musicbrainz    ident    map=mb_map
@@ -267,13 +269,13 @@ Creating the database
 
         local   all    all    trust
 
-    By default, the password for the user musicbrainz should be "musicbrainz",
-    as stated in lib/DBDefs.pm. You can change it with `psql`:
-
-        postgres=# ALTER USER musicbrainz UNENCRYPTED PASSWORD 'musicbrainz'
-
     Note that a running PostgreSQL will pick up changes to configuration files
-    only when being told so via a `HUP` signal.
+    only when being told so via a `HUP` signal (or by using pg_ctlcluster,
+    specifying `reload` as action).
+
+    You do not need to create the PostgreSQL user ("musicbrainz", or whatever
+    name you configured in DBDefs.pm) yourself; the next step will do so
+    (using the password from DBDefs.pm) if it does not exist yet.
 
 3.  Create the database
 
@@ -290,12 +292,12 @@ Creating the database
     2.  Import a database dump
 
         Our database dumps are provided twice a week and can be downloaded from
-        ftp://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/
+        [from a variety of locations](https://musicbrainz.org/doc/MusicBrainz_Database/Download#Download).
+        That page also describes the contents of the various dump files.
 
         To get going, you need at least the mbdump.tar.bz2,
         mbdump-editor.tar.bz2 and mbdump-derived.tar.bz2 archives, but you can
-        grab whichever dumps suit your needs. The online documentation has a
-        [description of the various data dump files](https://musicbrainz.org/doc/MusicBrainz_Database/Download#File_Descriptions).
+        grab whichever dumps suit your needs.
 
         Assuming the dumps have been downloaded to /tmp/dumps/ you can verify
         that the data is correct by running:
@@ -328,11 +330,16 @@ Creating the database
 
         If a full dump is too large for your purposes, but you would like to have some
         real data to test with for development, you can download our database sample,
-        published once a month at 
-        ftp://ftp.musicbrainz.org/pub/musicbrainz/data/sample/
-
+        published once a month. This can be found at the same places the full dump is
+        found (see above), but using `sample` instead of `fullexport` in the URL.
+        For example: http://ftp.musicbrainz.org/pub/musicbrainz/data/sample/.
+
         You can import this sample dump in the same way as the full dump above.
-
+
+    If this process gets interrupted or fails, you will need to manually drop the
+    musicbrainz_db database in order to be able to run `./admin/InitDb.pl --createdb`
+    again.
+
     MusicBrainz Server doesn't enforce any statement timeouts on any SQL it runs.
     If this is an issue in your setup, you may want to set a timeout at the
     database level:
@@ -380,14 +387,34 @@ If you intend to run a server with translations, there are a few steps to follow
 
 1.  Prerequisites
 
-    Make sure gettext is installed (you need msgmerge and msgfmt, at least),
-    and the transifex client 'tx'
-    (http://help.transifex.com/features/client/index.html):
+    Make sure gettext is installed (you need msgmerge and msgfmt, at least):
 
-        sudo apt-get install gettext transifex-client
+        sudo apt-get install gettext
+
+    This will enable you to compile and install the translations that are in
+    the source repository.
+
+    If you want to get the latest translation files or partial work-in-progress
+    translations, or wish to work on translations yourself, you will need to
+    create a [Transifex](https://www.transifex.com/) account and install its
+    client software (`tx`):
 
-    Configure a username and password in ~/.transifexrc using the format listed
-    on the above page.
+        sudo apt-get install transifex-client
+
+    More information (and alternative ways to install the client) can be found
+    [here](https://docs.transifex.com/client/introduction/).
+
+    Next, [create an API token](https://www.transifex.com/user/settings/api/)
+    and use it to configure your credentials in
+    [`~/.transifexrc`](https://docs.transifex.com/client/client-configuration#-transifexrc).
+
+    Finally, you will need to join the
+    [MetaBrainz Foundation organization](https://www.transifex.com/musicbrainz/public/)
+    on Transifex to get access to the translations. If you wish to work on
+    translations, you will also need to
+    [join a language team](https://www.transifex.com/musicbrainz/musicbrainz/dashboard/).
+    More information on how to get started can be found on
+    [the MusicBrainz site](https://musicbrainz.org/doc/Server_Internationalisation).
 
 2.  Change to the po directory
 
@@ -398,33 +425,32 @@ If you intend to run a server with translations, there are a few steps to follow
         tx pull -l {a list of languages you want to pull}
 
     This will download the .po files for your language(s) of choice to the po/
-    folder with the correct filenames.
+    folder with the correct filenames. Languages are written as an ISO language
+    code, optionally followed by an underscore and an ISO country code (e.g. `fr`
+    for French, `fr_CA` for Canadian French).
+
+    Or, if you want to get _all_ translations instead:
+
+        tx pull -a
+
+    If you get `Forbidden` errors from `tx pull`, you will need to make sure
+    you have joined the MusicBrainz organization and/or project (see point 1).
 
 4.  Install translations
 
         make install
 
     This will compile and install the files to
-    lib/LocaleData/{language}/LC\_MESSAGES/{domain}.mo
+    `lib/LocaleData/{language}/LC_MESSAGES/{domain}.mo`.
 
-5.  Add the languages to MB\_LANGUAGES in DBDefs.pm. These should be formatted
+5.  Add the languages to `MB_LANGUAGES` in DBDefs.pm. These should be formatted
     {lang}-{country}, e.g. 'es', or 'fr-ca', in a space-separated list.
 
-6.  Ensure you have a system locale for any languages you want to use, and for
-    some languages, be wary of https://rt.cpan.org/Public/Bug/Display.html?id=78341
-
-    For many languages, this will suffice:
+6.  Ensure you have a system locale for any languages you want to use. For many
+    languages, this will suffice:
 
         sudo apt-get install language-pack-{language code}
 
-    To work around the linked CPAN bug, you may need to edit the file for Locale::Util
-    to add entries to LANG2COUNTRY. Suggested ones include:
-
-    * es => 'ES'
-    * et => 'EE'
-    * el => 'GR'
-    * sl => 'SI' (this one is there in 1.20, but needs amendment)
-
 
 Troubleshooting
 ---------------

babel.config.js

@@ -37,6 +37,7 @@ module.exports = function (api) {
   const ignore = [
     'node_modules',
     'root/static/scripts/tests/typeInfo.js',
+    /root\/static\/build\/jed-[A-z_-]+?\.source\.js$/,
   ];
 
   return {

docker/Dockerfile.tests

@@ -64,6 +64,7 @@ RUN apt-get update && \
         openjdk-8-jre \
         openssh-client \
         perl \
+        pkg-config \
         postgresql \
         postgresql-10-pgtap \
         postgresql-contrib \

docker/templates/macros.m4

@@ -53,7 +53,8 @@ libicu-dev m4_dnl
 libperl-dev m4_dnl
 libpq-dev m4_dnl
 libssl-dev m4_dnl
-libxml2-dev')
+libxml2-dev m4_dnl
+pkg-config')
 
 # postgresql-server-dev-9.5 provides pg_config, which is needed by InitDb.pl
 # at run-time.

entities.json

@@ -199,6 +199,7 @@
         "removal": {
             "automatic": {}
         },
+        "series": true,
         "tags": true,
         "type": {
             "simple": true
@@ -483,6 +484,7 @@
             "manual": true
         },
         "report_filter": true,
+        "series": true,
         "sitemaps_lastmod_table": true,
         "tags": true,
         "url": "recording"
@@ -528,6 +530,7 @@
             "manual": true
         },
         "report_filter": true,
+        "series": true,
         "sitemaps_lastmod_table": true,
         "tags": true,
         "url": "release"
@@ -576,6 +579,7 @@
             }
         },
         "report_filter": true,
+        "series": true,
         "sitemaps_lastmod_table": true,
         "tags": true,
         "type": {
@@ -741,6 +745,7 @@
             "automatic": {}
         },
         "report_filter": true,
+        "series": true,
         "sitemaps_lastmod_table": true,
         "tags": true,
         "type": {

lib/DBDefs.pm.sample

@@ -374,6 +374,10 @@ sub WEB_SERVER                { "www.musicbrainz.example.com" }
 # sub COVER_ART_ARCHIVE_UPLOAD_PREFIXER { shift; sprintf("//%s.s3.us.archive.org/", shift) };
 # sub COVER_ART_ARCHIVE_DOWNLOAD_PREFIX { "//coverartarchive.org" };
 
+# Mapbox access token must be set to display area/place maps.
+# sub MAPBOX_MAP_ID { 'mapbox.streets' }
+# sub MAPBOX_ACCESS_TOKEN { '' }
+
 # Disallow OAuth2 requests over plain HTTP
 # sub OAUTH2_ENFORCE_TLS { my $self = shift; !$self->DB_STAGING_SERVER }
 

lib/DBDefs/Default.pm

@@ -358,6 +358,7 @@ sub COVER_ART_ARCHIVE_SECRET_KEY { };
 sub COVER_ART_ARCHIVE_UPLOAD_PREFIXER { shift; sprintf("//%s.s3.us.archive.org/", shift) };
 sub COVER_ART_ARCHIVE_DOWNLOAD_PREFIX { "//coverartarchive.org" };
 
+# Mapbox access token must be set to display area/place maps.
 sub MAPBOX_MAP_ID { 'mapbox.streets' }
 sub MAPBOX_ACCESS_TOKEN { '' }
 

lib/MusicBrainz/Server/Controller/Search.pm

@@ -178,6 +178,10 @@ sub direct : Private
             $c->model('Event')->load_related_info(@entities);
             $c->model('Event')->load_areas(@entities);
         }
+        when ('tag') {
+            #TODO: add support for genre aliases when finishing MBS-10062
+            $c->model('Genre')->load(@entities);
+        }
     }
 
     if ($type =~ /(recording|release|release_group)/)

lib/MusicBrainz/Server/Data/Artist.pm

@@ -313,7 +313,6 @@ sub merge
     $self->isni->merge($new_id, @$old_ids) unless is_special_artist($new_id);
     $self->tags->merge($new_id, @$old_ids);
     $self->rating->merge($new_id, @$old_ids);
-    $self->subscription->merge_entities($new_id, @$old_ids);
     $self->annotation->merge($new_id, @$old_ids);
     $self->c->model('ArtistCredit')->merge_artists($new_id, $old_ids, %opts);
     $self->c->model('Edit')->merge_entities('artist', $new_id, @$old_ids);

lib/MusicBrainz/Server/Data/Editor.pm

@@ -422,12 +422,20 @@ sub load_for_collection {
     return unless $id; # nothing to do
 
     $self->load($collection);
-    my $query = "SELECT " . $self->_columns . "
+    my $query = "SELECT " . $self->_columns . ", ep.value AS show_gravatar
                  FROM " . $self->_table . "
                  JOIN editor_collection_collaborator ecc ON editor.id = ecc.editor
+                 LEFT JOIN editor_preference ep ON ep.editor = editor.id AND ep.name = 'show_gravatar'
                  WHERE ecc.collection = $id
                  ORDER BY editor.name, editor.id";
-    my @collaborators = $self->query_to_list($query);
+    my @collaborators = $self->query_to_list($query, undef, sub {
+        my ($model, $row) = @_;
+
+        my $collaborator = $model->_new_from_row($row);
+        $collaborator->preferences->show_gravatar($row->{show_gravatar})
+            if defined $row->{show_gravatar};
+        $collaborator;
+    });
 
     $collection->collaborators(\@collaborators);
 }

lib/MusicBrainz/Server/Data/Label.pm

@@ -210,7 +210,6 @@ sub _merge_impl
     $self->isni->merge($new_id, @old_ids);
     $self->tags->merge($new_id, @old_ids);
     $self->rating->merge($new_id, @old_ids);
-    $self->subscription->merge_entities($new_id, @old_ids);
     $self->annotation->merge($new_id, @old_ids);
     $self->c->model('Collection')->merge_entities('label', $new_id, @old_ids);
     $self->c->model('ReleaseLabel')->merge_labels($new_id, @old_ids);

lib/MusicBrainz/Server/Data/Release.pm

@@ -647,7 +647,7 @@ sub _order_by {
         "date" => sub {
             return "date_year, date_month, date_day, musicbrainz_collate(name)"
         },
-        "title" => sub {
+        "name" => sub {
             return "musicbrainz_collate(name), date_year, date_month, date_day"
         },
         "country" => sub {

lib/MusicBrainz/Server/Data/Search.pm

@@ -301,9 +301,10 @@ sub search
 
     elsif ($type eq "tag") {
         $query = "
-            SELECT id, name, ts_rank_cd(to_tsvector('mb_simple', name), query, 2) AS rank
-            FROM tag, plainto_tsquery('mb_simple', ?) AS query
-            WHERE to_tsvector('mb_simple', name) @@ query OR name = ?
+            SELECT tag.id, tag.name, genre.id as genre_id,
+                   ts_rank_cd(to_tsvector('mb_simple', tag.name), query, 2) AS rank
+            FROM tag LEFT JOIN genre USING (name), plainto_tsquery('mb_simple', ?) AS query
+            WHERE to_tsvector('mb_simple', tag.name) @@ query OR tag.name = ?
             ORDER BY rank DESC, tag.name
             OFFSET ?
         ";

lib/MusicBrainz/Server/Data/Series.pm

@@ -93,7 +93,6 @@ sub _merge_impl {
 
     $self->alias->merge($new_id, @old_ids);
     $self->tags->merge($new_id, @old_ids);
-    $self->subscription->merge_entities($new_id, @old_ids);
     $self->annotation->merge($new_id, @old_ids);
     $self->c->model('Collection')->merge_entities('series', $new_id, @old_ids);
     $self->c->model('Edit')->merge_entities('series', $new_id, @old_ids);

lib/MusicBrainz/Server/Data/Subscription.pm

@@ -209,7 +209,7 @@ sub merge
               $edit_id, @ids);
 }
 
-sub merge_entities
+sub transfer_to_merge_target
 {
     my ($self, $new_id, @old_ids) = @_;
 

lib/MusicBrainz/Server/Data/URL.pm

@@ -48,6 +48,7 @@ my %URL_SPECIALIZATIONS = (
     'Dailymotion'         => qr{^https?://(?:www\.)?dailymotion\.com/}i,
     'DanceDB'             => qr{^https?://(?:www\.)?tedcrane\.com/DanceDB/}i,
     'Decoda'              => qr{^https?://(?:www\.)?decoda\.com/}i,
+    'Deezer'              => qr{^https?://(?:www\.)?deezer\.com/}i,
     'DHHU'                => qr{^https?://(?:www\.)?dhhu\.dk/}i,
     'Directlyrics'        => qr{^https?://(?:www\.)?directlyrics\.com/}i,
     'Discogs'             => qr{^https?://(?:www\.)?discogs\.com/}i,

lib/MusicBrainz/Server/Edit/Role/MergeSubscription.pm

@@ -9,6 +9,7 @@ requires 'subscription_model', '_entity_ids', 'do_merge', '_merge_model';
 around do_merge => sub {
     my ($orig, $self, @args) = @_;
 
+    $self->subscription_model->transfer_to_merge_target($self->new_entity->{id}, $self->_old_ids);
     my $editors = $self->subscription_model->delete($self->_old_ids);
     my $entities = $self->c->model($self->_merge_model)->get_by_ids($self->_old_ids);