Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP expanded Server side localization #2084

Closed

Conversation

Projects
None yet
4 participants
@buma
Copy link
Contributor

commented Aug 6, 2015

I started working on improved server localization as requested in #2070 .

First I replaced java properties localization with gettext resourceBundles since java properties doesn't support translation context and plural forms but gettext resourceBundles does. Both of this formats are automatically generated from gettext PO files.

I added long description to WalkSteps and some to leg.

I have some questions:

  • Are CreativeNamerEditor, NotePropertiesEditor and SafetyFeaturesEditor even needed anymore (since some of classes I needed to change are only used in creativeNamer)
  • Currently some translations uses sprintf format (%s, %d etc) and some named sprint format (%(streetname)s). I find sprint named format better for translators since it can be quickly seen what the parameter is without help of comments. But it needs conversion to normal sprintf format since Java doesn't support named parameters. I currently used same format that is already used in PO files.
  • Because translations are currently copied from client side they have HTML markup in them: Start on <b>Zrkovska cesta</b> heading east I think markup should stay in otherwise it is impossible to add in on the client but there should be an option to get text without the markup for OTP android for example.
  • It needs to be decided what text to return. Currently I used OTP.js strings on steps.

What is returned:

{
                        "agencyId": "MARPROM",
                        "agencyName": "Marprom",
                        "agencyTimeZoneOffset": 7200000,
                        "agencyUrl": "http://www.marprom.si/",
                        "arrivalDelay": 0,
                        "departureDelay": 0,
                        "distance": 3304.73980398542,
                        "duration": 1020.0,
                        "endTime": 1433850120000,
                        "from": {
                            "arrival": 1433848020000,
                            "departure": 1433849100000,
                            "lat": 46.5598401,
                            "lon": 15.6561071,
                            "name": "AP Mlinska",
                            "stopId": "MARPROM:68",
                            "stopIndex": 0,
                            "stopSequence": 1,
                            "vertexType": "TRANSIT"
                        },
                        "headsign": "Dupleška cesta 255",
                        "interlineWithPreviousLeg": false,
                        "legGeometry": {
                            "length": 249,
                            "points": "{ud{G_zp~A??vDdAvA\\@?NV@P?ZEj@E\\CXWG[ICAqA[GCq@UiAc@QGmCcAA?ICa@OLz@?DJt@PlABPBX@VBf@Dv@NhDAj@@bA@h@@\\BjA?B@d@BNBHBDBFJNJJNJNH^Hz@JxAHRLVHD?LBh@BV@RERANCHH@FBBDDD@Ct@G~AIpD@JBFHF\\Hx@NTBAB?B@@?DBB@@@@@?IdFAb@@P?H@HBVJn@BRB\\@l@?D?ZC@C@EDADADAH?D@F@DBDBBDBD?BABABCBC@E@C?EPDD?XBL@bEBpFHLFFDZLn@P^JPHJHLJNRPz@Nz@^fBLXFFH@DADCPW`@}CRuA`@kC@G@[?MCKEEECIAI?u@T]POF_@PWNQHG@E@WFQBQ?m@IMCIEKIEIGOHyBBq@FmA??@[?S?SAOG_@EOIOOWQYGGGQEOEQCQAS?M?m@@cA@wA?gD@iABgAB{@@o@FyANgD??PcDDsA?I@[FiNBqC@qCAsA??CyAEyBAi@Ai@Eu@EgA[gG]wFK{AEkAC_A?k@@u@FaA@MHi@D]Rw@??XmAP_AFk@Dc@?EDgA@iA?]AiE?g@EeR@sD?iADiAHs@H{@Lq@No@J_@L]qAs@Rw@Ny@^sB"
                        },
                        "mode": "BUS",
                        "pathway": false,
                        "realTime": false,
                        "rentedBike": false,
                        "route": "9",
                        "routeColor": "00FF01",
                        "routeId": "6",
                        "routeLongName": "Zrkovci",
                        "routeShortName": "9",
                        "routeTextColor": "FFFFFF",
                        "routeType": 3,
                        "serviceDate": "20150609",
                        "startTime": 1433849100000,
                        "steps": [],
                        "textDescription": "<b>9</b> Zrkovci to Dupleška cesta 255",
                        "to": {
                            "arrival": 1433850120000,
                            "departure": 1433850121000,
                            "lat": 46.5542438,
                            "lon": 15.6769781,
                            "name": "Zrkovska - GD",
                            "stopId": "MARPROM:119",
                            "stopIndex": 8,
                            "stopSequence": 10,
                            "vertexType": "TRANSIT"
                        },
                        "transitLeg": true,
                        "tripId": "327"
                    },
{
                        "agencyTimeZoneOffset": 7200000,
                        "arrivalDelay": 0,
                        "departureDelay": 0,
                        "distance": 148.60500000000002,
                        "duration": 116.0,
                        "endTime": 1433850237000,
                        "from": {
                            "arrival": 1433850120000,
                            "departure": 1433850121000,
                            "lat": 46.5542438,
                            "lon": 15.6769781,
                            "name": "Zrkovska - GD",
                            "stopId": "MARPROM:119",
                            "stopIndex": 8,
                            "stopSequence": 10,
                            "vertexType": "TRANSIT"
                        },
                        "interlineWithPreviousLeg": false,
                        "legGeometry": {
                            "length": 8,
                            "points": "gsc{Ge|t~ALw@V_BHk@Fc@Ji@Nk@Pa@"
                        },
                        "mode": "WALK",
                        "pathway": false,
                        "realTime": false,
                        "rentedBike": false,
                        "route": "",
                        "startTime": 1433850121000,
                        "steps": [
                            {
                                "absoluteDirection": "EAST",
                                "area": false,
                                "bogusName": false,
                                "distance": 148.60500000000002,
                                "elevation": [],
                                "lat": 46.55428198944995,
                                "lon": 15.676999344739935,
                                "longDescription": "Start on <b>Zrkovska cesta</b> heading east",
                                "relativeDirection": "DEPART",
                                "stayOn": false,
                                "streetName": "Zrkovska cesta"
                            }
                        ],
                        "textDescription": " to Zrkovska cesta",
                        "to": {
                            "arrival": 1433850237000,
                            "lat": 46.553772402865064,
                            "lon": 15.678750864151874,
                            "name": "Zrkovska cesta",
                            "orig": "",
                            "vertexType": "NORMAL"
                        },
                        "transitLeg": false
                    }
...
 {
                                "absoluteDirection": "NORTH",
                                "area": false,
                                "bogusName": true,
                                "distance": 176.33000000000004,
                                "elevation": [],
                                "lat": 46.5559524,
                                "lon": 15.6249359,
                                "longDescription": "<b>right</b> on to <b>bike path</b>",
                                "relativeDirection": "RIGHT",
                                "stayOn": false,
                                "streetName": "bike path"
                            },
                            {
                                "absoluteDirection": "NORTH",
                                "area": false,
                                "bogusName": true,
                                "distance": 62.384,
                                "elevation": [],
                                "lat": 46.5574255,
                                "lon": 15.6256863,
                                "longDescription": "<b>left</b> to continue on <b>path</b>",
                                "relativeDirection": "LEFT",
                                "stayOn": true,
                                "streetName": "path"
                            },
                            {
                                "absoluteDirection": "NORTH",
                                "area": false,
                                "bogusName": false,
                                "distance": 22.159,
                                "elevation": [],
                                "lat": 46.5574284,
                                "lon": 15.6261715,
                                "longDescription": "<b>slight left</b> on to <b>Ciril Metodova ulica</b>",
                                "relativeDirection": "SLIGHTLY_LEFT",
                                "stayOn": false,
                                "streetName": "Ciril Metodova ulica"
                            },

Currently it supports in walk step (which is used in walking, driving and cycling):

  • Start on %(streetName)s heading %(absoluteDirection)s
  • Take roundabout %(relativeDirection)s to %(ordinal_exit_number)s exit on %(streetName)s
  • if stay on is true %(relativeDirection)s to continue on %(streetName)s
  • %(relativeDirection)s on to %(streetName)s
    Missing are elevators and short text. Relative/absolute direction and ordinal exit numbers are localized.
    Currently there are translations for:
  • Catalan*
  • German*
  • Finnish
  • French*
  • Hungarian
  • Italian*
  • Dutch
  • Slovenian*
  • Swedish
    Those with * have at least some client side text translated (start on, take roundabout etc). The rest have only translations for generated street names, alerts, corners etc.

buma added some commits Aug 4, 2015

Mostly replace properties localization with gettext
All WayProperties and internals are replaced with Gettext Bundles.
GettextBundles are created from po files themselves.

Java properties don't support translation context and plurals.
That's why gettext bundles are used. To use them libintl library needs
to be included.

GettextLocalizedString and GettextCreative namer are probably
temporary. Currently Notes still use properties localization. Idea
 is to move all localization to Gettext bundles and server side.
Replace Alerts properties localization with Gettext
This replacement is made on all alerts that used java properties
localization. Currently notes and wheelchair description alert still
use another system. Since they use localization from OSM itself.

It needs to be decided how to improve this localization.
Add support for long walkstep server localization
Working on #2070
It adds longDescription and shortDescription field to WalkStep.
In longDescription is a long text description of a step:
- Start on [street name] heading [absolute direction]
- Take roundabout [clockwise/counter clockwise] to [ordinal exit
  number] exit on [street name]
- [Relative direction] to continue on [street name]
- [Relative direction] on to [street name]

ShortDescription is currently empty. It will be based on short text on
android OTP which is shown in map markers.

Relative direction are:
- Left, right, hard left, slight right, U-turn left, elevator etc.

Absolute direction are:
- north, south, east, west, northeast etc.

Currently it uses immutable maps for translations of ordinal exit
numbers and relative/absolute directions. All translatable strings are
used with T.tr function call which enables them to be automatically
extracted.
Localization of parameters (ordinal exit numbers, relative and absolute
direction) happens currently in WalkStep.

Message format ("Text {0} other text") is used to add parameters to
strings.

Note: Translations are enabled but don't exist yet in PO files or
Resource Bundles.
Add translations to descriptions in walk steps
Adds method which replaces sprintf named parameters to be used in Java
string formatting. And adds current client OTP.js po files.
This translations will be used to create text step descriptions.
Add text description to initenrary Legs
This is currently "route number route name to headsign/stop name" if
transit leg or "to destination name" if not
@barbeau

This comment has been minimized.

Copy link
Collaborator

commented Aug 6, 2015

@buma Thanks for working on this! Agreed, we'd need an option without markup for OTP Android, because parsing HTML on Android for display in TextViews can provide unpredictable results.

@abyrd

This comment has been minimized.

Copy link
Member

commented Aug 6, 2015

On sprintf versus named sprint parameters: Java does not have named sprint (i.e. where variables are passed in a String -> Object map and substituted into the string by name using the map). @buma thinks this will be easier for translators to understand in the future, so we should use it, and he has already made such a function. So everything will be switched to named-parameter sprint style.

@abyrd

This comment has been minimized.

Copy link
Member

commented Aug 6, 2015

About markup in returned translated strings: Our objective is to provide ready-to-use strings to the client that need to further processing. This text looks better with some items in bold, and just including the markup is as good a way as any to indicate that styling. A client could easily strip out the tags, but that is still extra work that needs to be duplicated across various clients. Agreed, the best option is to include an additional query parameter that strips out markup tags on the server side. This can easily be done with a regular expression.

@abyrd

This comment has been minimized.

Copy link
Member

commented Aug 6, 2015

Then we need to decide which text to return (current OTP text or current OTP.js text). In the case where one is shorter than the other, we will use one as the shortDescription and one as the longDescription. There would then be a shortDescription and longDescription in every leg and every step.This means manually merging the two translation files. We should do that first with the English files, make sure the whole system makes sense, then ask someone who speaks each language to perform the merge/re-translation once the list of keys is stable.

@abyrd

This comment has been minimized.

Copy link
Member

commented Aug 6, 2015

CreativeNamerEditor, NotePropertiesEditor and SafetyFeaturesEditor are all unused classes. They appear to be "bean" wrapper classes that were used in Spring XML configuration.

@buma

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2015

Differences between OTP.js and Opentripplaner strings:

  • Transit LEG
    • Opentripplanner: <b>transit mode:</b> agencyname, (route short name) route long name to headsign
    • OTP.js: link to agency name transit mode as an icon <b>route short name</b> route long name to destination stop/headsign
  • walk LEG
    • Opentripplanner: <b>WALK</b>: distance unit to destination name
    • OTP.js walk icon to destination name
  • cycling or driving is the same only WALK is replaced with CAR or BICYCLE and icon is different in otp.js

For non transit legs we can use Opentripplanner for long and OTP.js for short description. But for transit we need to decide what is better since they are basically the same length

@abyrd

This comment has been minimized.

Copy link
Member

commented Aug 7, 2015

I agree that we should re-use the existing OTP.js and OTP strings for the walk legs. But how will we include icons? It might be better to just make the short text "to destination", and let the client add an icon based on other fields in the response, or a link to the mode icon on the OTP server could be included in an HTML tag.

Maybe the short form of transit legs should be just the route_short_name with no other text.

Another point: in the English base text, the preposition before the headsign should be "toward" rather than "to". "To" implies that you will get off at that final stop.

@buma

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2015

One way to add an icon is to add a parameter %(icon) or and client can then replace it with correct icon.

Add short and long description to Leg translations
Short description for Transit Legs is currently missing.
@johannilsson

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2015

I'm late to this discussion. We're maintaining clients for OTP and handle all translations within our clients. As long as current constants is not changed providing a set of base translations is totally fine and I can see this benefiting the official clients for this project.

I however don't see the benefit of the server telling the clients how the data should be presented with e.g. markup and or indicators for where an icon should be placed. The current translations also makes assumptions on how the data is supposed to be presented, for example should the first character for the message be uppercased or not? If providing translations from the server I think it's important that they can work in any context and not making assumptions on how it's presented.

@barbeau

This comment has been minimized.

Copy link
Collaborator

commented Aug 7, 2015

I agree with @johannilsson - we'd have to strip out this icon info on OTP Android, so I'd prefer not to have it included - or, similar to markup, have strings with and without the info (e.g., @abyrd's suggestion of link to the mode icon on the OTP server being included in an HTML tag). Android has a fairly complex way of presenting different icons due to different screen sizes and densities, and we tend to also do some color filtering and dynamic resizing to theme icons to different colors/places within the app. And placement of the icon in context with the directions may differ based on screen size. As a result these icons really need to be bundled with the app, or we'd have to manually do much of the above which the Android platform currently handles itself.

On a different topic, would server-side localization include currency support for fares too?

@abyrd

This comment has been minimized.

Copy link
Member

commented Aug 7, 2015

My initial preference was to not include any markup, icons or anything in the translated text. The problem is that some people think the bold/formatted text looks better, and it's impossible to know which part of the message to format when it's returned as a monolithic string. So the only way is to include markup in the monolithic string and allow the markup to be turned off.

Including tags that reference icons is pushing it though. I think we should just include translated text, with formatting tags if users find that increases readability.

@abyrd

This comment has been minimized.

Copy link
Member

commented Aug 7, 2015

@barbeau the fare currency is defined in GTFS using https://en.wikipedia.org/wiki/ISO_4217 currency codes.

@barbeau

This comment has been minimized.

Copy link
Collaborator

commented Aug 7, 2015

@abyrd Ok, great.

@johannilsson

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2015

Yeah I can live with the markup being there too. However I would suggest making the strings them self work independent of how they're presented. I would guess this would help potential screen readers as well. If a string presented in the instruction only reads as "to destination" this wouldn't give much help. Strings that's cut of and "requires" another part usually leads to problem when dealing with RTL languages as well, but yes it can of course be resolved.

@abyrd

This comment has been minimized.

Copy link
Member

commented Aug 7, 2015

@johannilsson I think that's the idea of the short and long strings. The long one will always be perfectly clear stand-alone, the short one might make some compromises to be short.

@abyrd

This comment has been minimized.

Copy link
Member

commented Aug 7, 2015

Also remember that there will be a query parameter to turn off the markup tags.

buma added some commits Aug 10, 2015

Remove duplicate LocalizedString and CreativeNamer classes
GettextCreativeNamer and GettextLocalizedString are renamed to
CreativeNamer and LocalizedString since Gettext is always used.

For notes that don't use Gettext localization (note pattern like:
"{note}", "{notes}", "{wheelchair:description}") use localization from
OSM and are excluded from current changes. If in the future we need
some texts which uses Tag localization from OSM and Gettext
localization it would need to be added.
Replace unnamed printf parameters with named in translations
LocalizedStrings has now three constructors:
- first with translatable object and map of parameter names and values
- second with translatable object and OSM way
- third with just translatable object (this is used when no parameters
  are used in translation)
Replace string key with translatable key in LocalizedString
Translatable key supports message and context and is better for future
extensions. (plural if needed)

buma added some commits Aug 13, 2015

Add localization for distance strings in Leg
Units can be metric or Imperial. They are set based on Locale.
Currently English has imperial units and everything else is in Metric.
Change text from to -> towards in Leg description
Routename to headsign was wrong. Correct is Routename towards headsign.
"to" is still used for last station and destination name.
@buma

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2015

I looked into OTP Android and it uses those strings for short versions:

  • Route route short name/last 12 characters of route long name from leg.from.name
  • Walk to leg.to.name
  • Ride to leg.to.name

I didn't find if it shows anything on steps itself. @barbeau can you please tell if OTP Android uses any short versions for steps itself? (turn left, right, etc.)

OTP.js uses turn icon and mode icon and destination name on hover on turns. But this can be combined on the client itself.

@buma

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2015

Since some client use icons and some don't and use multiple different short/long text another thing to do would be to return keys and translated string and ideas of how to put them together and then it is up to client to combine them.

One option is to return dictionary where keys are what is currently returned in relative/absolute direction and exit. Values are translated keys.
Another option is to add localized_exit, relative and absolute direction to step itself.
And then in dictionary only add suggested short/long leg and step string.
Clients can then use suggested strings or just use translated directions and use own string.
I think this is something that @johannilsson and also @abyrd had in mind.

Something like this (numbers are translated exit numbers, then there are relative and absolute directions)
For step a suggested order is made:
"STEP_CONTINUE_STAY_ON": "<b>%(relativeDirection)s</b> nadaljujte na <b>%(streetName)s</b>",
Parameter names are same names as are in Step/Leg itself. So in Javascript with sprintf translation could look like: `

//this function would localize Relative/Absolute direction and exit numbers
localized_current_step =localize(current_step);
sprintf(translations["STEP_CONTINUE_STAY_ON"],localized_current_step)
//would return "<b>rahlo levo</b> nadaljujte na <b>Sokolska ulica</b>"
"translations": {
                                    "1": "prvem",
                                    "10": "desetem",
                                    "2": "drugem",
                                    "3": "tretjem",
                                    "4": "četrtem",
                                    "5": "petem",
                                    "6": "šestem",
                                    "7": "sedmem",
                                    "8": "osmem",
                                    "9": "devetem",
                                    "CIRCLE_CLOCKWISE": "v smeri urinega kazalca",
                                    "CIRCLE_COUNTERCLOCKWISE": "v nasprotni smeri urinega kazalca",
                                    "CONTINUE": "nadaljujte",
                                    "DEPART": "začni pot",
                                    "EAST": "vzhod",
                                    "ELEVATOR": "pojdite z dvigalom",
                                    "HARD_LEFT": "ostro levo",
                                    "HARD_RIGHT": "ostro desno",
                                    "LEFT": "levo",
                                    "LEG_SHORT_TRANSIT": "Linija %(routeShortName)s od %(from.name)s",
                                    "NORTH": "sever",
                                    "NORTHEAST": "severovzhod",
                                    "NORTHWEST": "severozahod",
                                    "RIGHT": "desno",
                                    "SLIGHTLY_LEFT": "rahlo levo",
                                    "SLIGHTLY_RIGHT": "rahlo desno",
                                    "SOUTH": "jug",
                                    "SOUTHEAST": "jugovzhod",
                                    "SOUTHWEST": "jugozahod",
                                    "STEP_CONTINUE": "<b>%(relativeDirection)s</b> na <b>%(streetName)s</b>",
                                    "STEP_CONTINUE_STAY_ON": "<b>%(relativeDirection)s</b> nadaljujte na <b>%(streetName)s</b>",
                                    "STEP_ROUNDABOUT": "V krožišču vozite %(relativeDirection)s in pri %(exit)s izvozu zavijte na %(streetName)s",
                                    "STEP_START": "Začnite na <b>%(streetName)s</b> v smeri %(absoluteDirection)s",
                                    "UTURN_LEFT": "Polkrožno obrnite v levo",
                                    "UTURN_RIGHT": "Polkrožno obrnite v desno",
                                    "WEST": "zahod"
                                }
@barbeau

This comment has been minimized.

Copy link
Collaborator

commented Aug 17, 2015

@buma I don't believe we're using the short versions of the steps themselves in OTP Android.

@abyrd

This comment has been minimized.

Copy link
Member

commented Aug 22, 2015

@buma do you still consider this work in progress or is this advanced enough to merge and use in its current state?

@buma

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2015

Creation of java class files and extraction of translations scripts are still missing.

@abyrd

This comment has been minimized.

Copy link
Member

commented Jul 14, 2016

It looks like development on this branch is suspended and is not intended to make it into 1.0 release. @buma please let me know if I've misunderstood!

@abyrd abyrd closed this Jul 14, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.