Add TM to the Angular editor. Adds a REST endpoint.

Squashed commit of the following:

commit 81f4476
Author: David Mason <drdmason@gmail.com>
Date:   Thu Jun 11 10:32:06 2015 +1000

    Remove fallback to stop-words comparison for similarity score calculation.

    This change is contentious, so I am removing it until an amicable resoultion
    to the discussion is reached. Similarity scores need more user testing to
    determine which algorithm will b emost useful to translators.

commit 882e301
Author: David Mason <drdmason@gmail.com>
Date:   Thu Jun 11 10:19:01 2015 +1000

    Revert "Change token regex for similarity comparison to pick up more cases."

    This reverts commit db71115.

    More test-cases are required to ensure this regex is correct. Rolling back for
    now so that this change does not go into the next release without thorough
    testing.

    The main problem with the new regex is that it would break on '.' in constructions
    such as URLs, which does not appear to be desired behaviour.

commit 70cbb12
Author: David Mason <drdmason@gmail.com>
Date:   Tue Jun 9 13:59:05 2015 +1000

    Update zanata-spa to commit 883b7b9071ef42c5e021c62905d311e985ec8596

commit db71115
Author: David Mason <drdmason@gmail.com>
Date:   Tue Jun 9 13:38:10 2015 +1000

    Change token regex for similarity comparison to pick up more cases.

    The regex was not picking up punctuation that was not followed by whitespace.
    There are still some cases that it is not covering. I have kept this change
    small, and the other changes can be made when there is more time for testing.

commit 276f310
Author: David Mason <drdmason@gmail.com>
Date:   Tue Jun 9 08:53:54 2015 +1000

    Fall back on stop-words comparison for similarity when nothing else is available.

    A divide-by-zero condition was generating a NaN value for similarity for strings
    that contain only stop-words. When that condition would be met, this will instead
    repeat the comparison without ignoring stop-words, then finally just return 0.0
    if there are still no tokens to compare.

commit 35e0e53
Author: David Mason <drdmason@gmail.com>
Date:   Tue Jun 9 13:10:30 2015 +1000

    Log more information when TM result row has an unexpected entity type.

    When lucene indexes are in a bad state, search can return a row that
    does not contain one of the expected entity types. This ensures that
    valid rows will still be returned in results, and an error with more
    information will be logged when a bad row is encountered.

commit 0144585
Author: David Mason <drdmason@gmail.com>
Date:   Thu Jun 4 16:12:07 2015 +1000

    Update zanata-spa to commit 6f20797323e6d45d05eff01cfefb0969f50e5479

commit 990e0ad
Author: David Mason <drdmason@gmail.com>
Date:   Thu Jun 4 13:11:51 2015 +1000

    Update zanata-spa to commit 194a5e237ff5edd47953b89e2e31255de67d0a1d

commit c20e1b7
Author: David Mason <drdmason@gmail.com>
Date:   Thu May 21 20:58:11 2015 +1000

    Update zanata-spa to commit 402c57d0ebff0ea957f31e4a4e508974b89d9099

commit 68b2b27
Author: David Mason <drdmason@gmail.com>
Date:   Thu May 21 15:05:15 2015 +1000

    Add release note for suggestions endpoint.

commit f643587
Author: David Mason <drdmason@gmail.com>
Date:   Wed May 20 08:36:50 2015 +1000

    Refactor LocaleServiceImpl.getByLocaleId(String) for brevity.

commit 0a4e2c5
Author: David Mason <drdmason@gmail.com>
Date:   Tue May 19 11:34:44 2015 +1000

    Change source and translation locales for suggestions endpoint to query params.

    Since locales can contain non-url-friendly characters, they are better included
    as part of the query string.

commit d98a2fa
Author: David Mason <drdmason@gmail.com>
Date:   Tue May 19 14:59:05 2015 +1000

    Remove unused media type string for suggestions.

    Changed to use the JSON media type for suggestions.

commit 36bf6b7
Author: David Mason <drdmason@gmail.com>
Date:   Tue May 19 14:23:32 2015 +1000

    Use Option instead of Either in SuggestionService

commit 7d65aea
Author: David Mason <drdmason@gmail.com>
Date:   Tue May 19 14:22:43 2015 +1000

    Add @nullable annotation to locale lookup.

commit 0306816
Author: David Mason <drdmason@gmail.com>
Date:   Tue May 19 11:01:23 2015 +1000

    Prevent undeclared IllegalArgumentException when looking up locales.

commit 68a25b8
Author: David Mason <drdmason@gmail.com>
Date:   Tue May 19 10:37:10 2015 +1000

    Make search type lookup case-insensitive.

    The try block could be removed because iterating enum values will not
    give nulls, and equalsIgnoreCase will behave sensibly with an entered
    null value.

commit 0b8e828
Author: David Mason <drdmason@gmail.com>
Date:   Tue May 19 10:24:11 2015 +1000

    Keep format strings in the same scope as String.format()

    Format strings were being passed to a function, which seems confusing
    and so more error-prone. Changed to passing an already-formatted string
    so that the format strings can be inlined in the call to String.format()
    so that it is obvious that they are format strings and what the
    placeholders represent.

commit 727c0a0
Author: David Mason <drdmason@gmail.com>
Date:   Sat May 9 22:09:39 2015 +1000

    Add REST endpoint for suggestions for editor.

commit 3b8addd
Author: David Mason <drdmason@gmail.com>
Date:   Fri May 15 21:34:36 2015 +1000

    Add method to get suggestions from TM service.

    I have avoided changing the existing code paths for TM search for now
    since it is a performance-critical component for translation copy. We
    should combine the code paths in a separate operation as long as it
    does not impact on performance.

commit bee070e
Author: David Mason <drdmason@gmail.com>
Date:   Fri May 15 22:34:51 2015 +1000

    Add DTOs for suggestions and suggestion detail.

    These DTO classes are only configured to serialize to JSON, and I have
    not considered deserialization since this is query data and not expected
    to be uploaded to the server.

commit a53c3cc
Author: David Mason <drdmason@gmail.com>
Date:   Wed May 13 08:58:52 2015 +1000

    Workaround TM result coalescing bug.

    Refactors the match processing code to make it clearer what type of
    match is being added to a result item, and makes a few small changes
    to minimize the incorrect behaviour:

     - No longer adds imported TM item id to the list that is used to look
       up text flows.

           A more complete fix is to wrap the Long id in a type so that
           all the ids are present and can be used sensibly. This just
           makes the list appropriate for the code that uses it right
           now, to delay spending the development time for the full fix.

     - Upgrades match type when a match has a higher match type, where
       ApprovedInternal > TranslatedInternal > Imported.

           Imported is treated as lowest just because the editor suppresses
           details display for Imported. Treating the others as higher will
           allow users to see text flow details for the other types.

           The full fix for this would be to inculde a summary of all the
           match types in the DTO, so the editor can display whatever
           makes sense.

docs/release-notes.md

@@ -68,6 +68,8 @@ Example usage in html file: `<link rel="shortcut icon" href="#{assets['img/logo/
 * [1204982](https://bugzilla.redhat.com/show_bug.cgi?id=1204982) - Documentation update for zanata.org/help + readthedocs
 * [1211849](https://bugzilla.redhat.com/show_bug.cgi?id=1211849) - Project maintainer can change project/version slug
 * [1082840](https://bugzilla.redhat.com/show_bug.cgi?id=1082840) - Project maintainer can delete a project or project version
+* [1209669](https://bugzilla.redhat.com/show_bug.cgi?id=1209669) - New REST endpoint for editor suggestions.
+*
 
 ## 3.6.3
 

zanata-war/src/main/java/org/zanata/rest/editor/MediaTypes.java

@@ -32,6 +32,9 @@ public String toString() {
     public static final String APPLICATION_ZANATA_LOCALES_JSON =
             APPLICATION_ZANATA_LOCALES + JSON;
 
+    public static final String APPLICATION_ZANATA_SUGGESTIONS_JSON =
+            APPLICATION_VND_ZANATA + ".suggestions" + JSON;
+
     public static final String APPLICATION_ZANATA_PROJECT_VERSION =
             APPLICATION_VND_ZANATA + ".version";
     public static final String APPLICATION_ZANATA_PROJECT_VERSION_JSON =

zanata-war/src/main/java/org/zanata/rest/editor/dto/suggestion/JsonDateSerializer.java

@@ -0,0 +1,49 @@
+/*
+ * Copyright 2015, Red Hat, Inc. and individual contributors
+ * as indicated by the @author tags. See the copyright.txt file in the
+ * distribution for a full listing of individual contributors.
+ *
+ * This is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU Lesser General Public License as
+ * published by the Free Software Foundation; either version 2.1 of
+ * the License, or (at your option) any later version.
+ *
+ * This software is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this software; if not, write to the Free
+ * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
+ * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
+ */
+package org.zanata.rest.editor.dto.suggestion;
+
+import org.codehaus.jackson.JsonGenerator;
+import org.codehaus.jackson.map.JsonSerializer;
+import org.codehaus.jackson.map.SerializerProvider;
+import org.joda.time.DateTime;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
+
+import java.io.IOException;
+import java.util.Date;
+
+/**
+ * Serializer to output dates in ISO-8601 format.
+ *
+ * This format is used for the JSON API because it is compatible with
+ * JavaScript Date.parse() and is a widely used standard.
+ */
+public class JsonDateSerializer extends JsonSerializer<Date> {
+
+    private static final DateTimeFormatter ISO8601Format = ISODateTimeFormat.dateTime();
+
+    @Override
+    public void serialize(Date date, JsonGenerator jsonGenerator, SerializerProvider provider) throws IOException {
+        String dateString = ISO8601Format.print(new DateTime(date));
+        jsonGenerator.writeString(dateString);
+    }
+
+}

zanata-war/src/main/java/org/zanata/rest/editor/dto/suggestion/Suggestion.java

@@ -0,0 +1,59 @@
+/*
+ * Copyright 2015, Red Hat, Inc. and individual contributors
+ * as indicated by the @author tags. See the copyright.txt file in the
+ * distribution for a full listing of individual contributors.
+ *
+ * This is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU Lesser General Public License as
+ * published by the Free Software Foundation; either version 2.1 of
+ * the License, or (at your option) any later version.
+ *
+ * This software is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this software; if not, write to the Free
+ * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
+ * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
+ */
+package org.zanata.rest.editor.dto.suggestion;
+
+import lombok.Getter;
+import org.codehaus.jackson.annotate.JsonPropertyOrder;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Represents a single suggested translation.
+ *
+ * This could be from translation memory or other sources.
+ *
+ * This representation is designed for use with the pure JavaScript editor.
+ */
+@Getter
+@JsonPropertyOrder({ "relevanceScore", "similarityPercent", "sourceContents", "targetContents", "matchDetails" })
+@JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+public class Suggestion implements Serializable {
+
+    private final double relevanceScore;
+    private final double similarityPercent;
+
+    private final List<String> sourceContents;
+    private final List<String> targetContents;
+
+    private final List<SuggestionDetail> matchDetails;
+
+    public Suggestion(double relevanceScore, double similarityPercent,
+                      List<String> sourceContents, List<String> targetContents) {
+        this.relevanceScore = relevanceScore;
+        this.similarityPercent = similarityPercent;
+        this.sourceContents = sourceContents;
+        this.targetContents = targetContents;
+        this.matchDetails = new ArrayList<>();
+    }
+}

zanata-war/src/main/java/org/zanata/rest/editor/dto/suggestion/SuggestionDetail.java

@@ -0,0 +1,53 @@
+/*
+ * Copyright 2015, Red Hat, Inc. and individual contributors
+ * as indicated by the @author tags. See the copyright.txt file in the
+ * distribution for a full listing of individual contributors.
+ *
+ * This is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU Lesser General Public License as
+ * published by the Free Software Foundation; either version 2.1 of
+ * the License, or (at your option) any later version.
+ *
+ * This software is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this software; if not, write to the Free
+ * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
+ * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
+ */
+package org.zanata.rest.editor.dto.suggestion;
+
+import java.io.Serializable;
+
+/**
+ * Detailed information about a suggestion of a specific type.
+ */
+public interface SuggestionDetail extends Serializable {
+
+    /**
+     * Possible types of suggestions from different resources.
+     *
+     * Different types may present different information, so use
+     * different class representations.
+     */
+    enum SuggestionType {
+
+        /**
+         * A suggestion from a project on this Zanata server.
+         */
+        LOCAL_PROJECT,
+
+        /**
+         * A suggestion from an imported translation memory.
+         */
+        IMPORTED_TM
+    }
+
+    /**
+     * @return the type of suggestion.
+     */
+    SuggestionType getType();
+}

zanata-war/src/main/java/org/zanata/rest/editor/dto/suggestion/TextFlowSuggestionDetail.java

@@ -0,0 +1,91 @@
+/*
+ * Copyright 2015, Red Hat, Inc. and individual contributors
+ * as indicated by the @author tags. See the copyright.txt file in the
+ * distribution for a full listing of individual contributors.
+ *
+ * This is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU Lesser General Public License as
+ * published by the Free Software Foundation; either version 2.1 of
+ * the License, or (at your option) any later version.
+ *
+ * This software is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this software; if not, write to the Free
+ * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
+ * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
+ */
+package org.zanata.rest.editor.dto.suggestion;
+
+import lombok.Getter;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+import org.zanata.common.ContentState;
+import org.zanata.model.*;
+
+import java.util.Date;
+
+/**
+ * Detailed information about a suggestion from a project on this server.
+ */
+@Getter
+@JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+public class TextFlowSuggestionDetail implements SuggestionDetail {
+
+    private final SuggestionType type = SuggestionType.LOCAL_PROJECT;
+
+    private final Long textFlowId;
+
+    private final String sourceComment;
+    private final String targetComment;
+
+    private final ContentState contentState;
+
+    private final String projectId;
+    private final String projectName;
+    private final String version;
+    private final String documentName;
+    private final String documentPath;
+
+    private final String resId;
+
+    // TODO use @JsonFormat for date format when jackson is updated to 2+
+    @JsonSerialize(using = JsonDateSerializer.class)
+    private final Date lastModifiedDate;
+    private final String lastModifiedBy;
+
+    /**
+     * Create a detail object based on a given text flow target.
+     *
+     * @param tft for which to create a detail object.
+     */
+    public TextFlowSuggestionDetail(HTextFlowTarget tft) {
+        HTextFlow tf = tft.getTextFlow();
+        final HDocument document = tf.getDocument();
+        final HProjectIteration version = document.getProjectIteration();
+        final HProject project = version.getProject();
+        final HPerson lastModifiedPerson = tft.getLastModifiedBy();
+        final boolean haveLastModifiedUsername = lastModifiedPerson != null && lastModifiedPerson.hasAccount();
+
+        this.textFlowId = tf.getId();
+
+        this.sourceComment = HSimpleComment.toString(tf.getComment());
+        this.targetComment = HSimpleComment.toString(tft.getComment());
+
+        this.contentState = tft.getState();
+
+        this.projectId = project.getSlug();
+        this.projectName = project.getName();
+        this.version = version.getSlug();
+        this.documentName = document.getName();
+        this.documentPath = document.getPath();
+        this.resId = tf.getResId();
+
+        this.lastModifiedDate = tft.getLastChanged();
+        this.lastModifiedBy = haveLastModifiedUsername ?
+                lastModifiedPerson.getAccount().getUsername() : null;
+    }
+
+}

zanata-war/src/main/java/org/zanata/rest/editor/dto/suggestion/TransMemoryUnitSuggestionDetail.java

@@ -0,0 +1,60 @@
+/*
+ * Copyright 2015, Red Hat, Inc. and individual contributors
+ * as indicated by the @author tags. See the copyright.txt file in the
+ * distribution for a full listing of individual contributors.
+ *
+ * This is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU Lesser General Public License as
+ * published by the Free Software Foundation; either version 2.1 of
+ * the License, or (at your option) any later version.
+ *
+ * This software is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this software; if not, write to the Free
+ * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
+ * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
+ */
+package org.zanata.rest.editor.dto.suggestion;
+
+import lombok.Getter;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+import org.zanata.model.tm.TransMemoryUnit;
+
+import java.util.Date;
+
+/**
+ * Detailed information about a suggestion from an imported translation memory.
+ */
+@Getter
+@JsonSerialize(include = JsonSerialize.Inclusion.NON_NULL)
+public class TransMemoryUnitSuggestionDetail implements SuggestionDetail {
+
+    private final SuggestionType type = SuggestionType.IMPORTED_TM;
+
+    /**
+     * The database id that can be used to look up the TransMemoryUnit.
+     */
+    private final Long transMemoryUnitId;
+
+    private final String transMemorySlug;
+    private final String transUnitId;
+
+    @JsonSerialize(using = JsonDateSerializer.class)
+    private final Date lastChanged;
+
+    /**
+     * Create a detail object based on a given trans memory unit.
+     *
+     * @param tmUnit for which to create a detail object
+     */
+    public TransMemoryUnitSuggestionDetail(TransMemoryUnit tmUnit) {
+        this.transMemoryUnitId = tmUnit.getId();
+        this.transMemorySlug = tmUnit.getTranslationMemory().getSlug();
+        this.transUnitId = tmUnit.getTransUnitId();
+        this.lastChanged = tmUnit.getLastChanged();
+    }
+}