diff --git a/README.md b/README.md index 2f8cf1a761..c748d1f87e 100644 --- a/README.md +++ b/README.md @@ -20,13 +20,14 @@ Test Pilot is not intended to replace trains for most features, nor is it a test - [Verifying deployments](docs/development/verification.md) - Verifying Test Pilot deployments. - [Add-on environment](docs/development/environment.md) - Configuring to which server the add-on connects. - Developing experiments + - [Experiment metrics](docs/experiments/ga.md) - The use of Google Analytics to track experiment data. - [Variant testing](docs/experiments/variants.md) - Creating variant (e.g. A/B) tests in experiments. - [Example experiments](docs/experiments/) - Metrics - [Telemetry](docs/metrics/telemetry.md) - How we use Firefox telemetry. - [Google Analytics](docs/metrics/ga.md) - How we use Google Analytics. - [New features](docs/metrics/new_features.md) - Everything needed to instrument something new. -- [Experiment content] (docs/content/reference.md) - Management of experiment content. +- [Experiment content](docs/content/reference.md) - Management of experiment content. - [Process](docs/process.md) - How we create, triage, and assign work. - [FAQ](docs/faq.md) - [Contributing to Test Pilot](CONTRIBUTING.md) diff --git a/docs/experiments/ga.md b/docs/experiments/ga.md new file mode 100644 index 0000000000..f0d2c55b7f --- /dev/null +++ b/docs/experiments/ga.md @@ -0,0 +1,67 @@ +[👈 Back to README](../../README.md) + +# Experiment Metrics +Test Pilot experiments use Google Analytics for metrics collection, and Google Data Studio for visualization and reporting. Each experiment should create and use a new property in Mozilla’s Google Analytics account. If you need help doing so, please talk to Wil Clouser. + +Events are reported through the low-level [Google Analytics Measurement Protocol](https://developers.google.com/analytics/devguides/collection/protocol/v1/). Refer to the documentation for the [developer guide](https://developers.google.com/analytics/devguides/collection/protocol/v1/devguide) and [parameter reference](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters) to understand to basics of how data is reported. The [hit builder](https://ga-dev-tools.appspot.com/hit-builder/) can help you construct and validate events before reporting. + +## Submission +The following fields are used in experiment event reporting. All fields are required, unless noted. Experiments should implement additional properties of [sessions](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#session), [exceptions](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#exception), [social interactions](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#social), [traffic sources](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#trafficsources), [content information](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#content), and [system attributes](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#system), where appropriate. + +### General Fields +- [`v`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#v) - this indicates the version of the measurement protocol being used. Currently always `1`. +- [`tid`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#tid) - this indicates the ID of the Google Analytics property being used by the experiment. Should follow the form `UA-XXXX-Y`. +- [`aip`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#aip) - this indicates that the IP address of the sender should be anonymized. Always `1`. +- [`ds`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#ds) - this indicates the context from which the event is being reported. Should be `addon` if sent from an experiment’s add-on, `web` if sent from an associated web property, or `app` if sent from an associated mobile application. +- [`qt`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#qt) - if [batching](https://developers.google.com/analytics/devguides/collection/protocol/v1/devguide#batch) or delaying reports, this should represent the time delta between the event happening and the time it is being reported. This should not exceed 4 hours. _(Optional)_ +- [`z`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#z) - the unix timestamp of the event taking place, used for cache-busting. + +### User Fields +- [`cid`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#cid) - a [UUIDv4](https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_.28random.29) that should be consistent across all events reported with this user. It should be stored with a persistent mechanism. If there is a sync component to the experiment, this should also be synced and be made consistent across clients and devices. This can be generated with [the `uuid` npm package](https://www.npmjs.com/package/uuid). +- [`uid`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#uid) - the user’s telemetry ID, available at `toolkit.telemetry.cachedClientID`. _(Optional)_ + +### Session +- [`ua`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#ua) - the user agent, as reported by [`navigator.userAgent`](https://developer.mozilla.org/en-US/docs/Web/API/NavigatorID/userAgent). + +### System Info +- [`ul`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#ul) - the user’s language, as reported by [`navigator.language`](https://developer.mozilla.org/en-US/docs/Web/API/NavigatorLanguage/language). + +### Hit +- [`t`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#t) - the type of hit. Usually `event` for Test Pilot usage. + +### App Tracking Fields +- [`an`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#an) - the name of the Test Pilot experiment. +- [`aid`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#aid) - the add-on’s ID, if one exists. _(Optional)_ +- [`av`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#av) - the add-on’s version number, if one exists. _(Optional)_ +- [`aiid`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#aiid) - an indication of the location of the experiment. For most usage, should be `testpilot`. This should be changed if shipping with [Shield](https://wiki.mozilla.org/Firefox/Shield) or in moz-central. + +### Event Fields +For more information about the components of an event and best practices for doing so, see [Google’s documentation on events](https://support.google.com/analytics/answer/1033068) . + +- [`ec`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#ec) - the category of the event being reported. +- [`ea`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#ea) - the action of the event being reported. +- [`el`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#el) - the event’s label. _(Optional)_ +- [`ev`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#ev) - the event’s value, as an integer. _(Optional)_ + +### Custom Dimension/Metric Fields +Google Analytics allows you to report [custom dimensions and metrics](https://support.google.com/analytics/answer/2709828?hl=en) with each event, up to 20 of each. Care should be taken in choosing these, given their limited number and inability to repurpose them once used. They are [defined in the property’s admin](https://support.google.com/analytics/answer/2709828?hl=en#configuration) before being used, and their definition and use are documented in each experiment’s repository. + +- [`cd`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#cd_) - the value of custom dimension number ``. +- [`cm`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#cm_) - the value of custom metric number ``. + +#### Global Custom Dimensions/Metrics +Some custom dimensions and metrics are used consistently across Test Pilot experiments. These use the highest index numbers possible, for consistency across reports. These are all required. + +- `cd20` - indicates the Firefox release channel. This can be reverse-engineered by comparing the user agent to [information from `product-details`](https://product-details.mozilla.org/1.0/). Should be one of `esr`, `release`, `beta`, `developer`, or `nightly`. + +### Content Experiment Fields +- [`xid`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#xid) - the ID of the current experiment._(Optional)_ +- [`xvar`](https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters#xvar) - the name of the population to which the user belongs in the current experiment. _(Optional)_ + +## Content Experiments +By using the [content experiment fields](#content-experiment-fields), experiments are free to use Google Analytics’ [Content Experiments](https://support.google.com/analytics/answer/1745147?hl=en&ref_topic=1745207&visit_id=1-636335713647262387-4167257197&rd=1) for variant testing. Users may only belong to one experiment at a time. + +Eventually, this section will include more specific information about the use of content experiments. See [bug 2537](https://github.com/mozilla/testpilot/issues/2537) for details. + +## Visualization and Reporting +Eventually, this section will include information about the use of Data Studio to analyze and visualize collected data. See [bug 2539](https://github.com/mozilla/testpilot/issues/2539) for details. diff --git a/locales/es-AR/app.ftl b/locales/es-AR/app.ftl index 60ecd76d10..fc19c86734 100644 --- a/locales/es-AR/app.ftl +++ b/locales/es-AR/app.ftl @@ -101,6 +101,12 @@ emailOptInSuccessMessage2 = ¡Gracias! emailOptInConfirmationClose = Hacia los experimentos... +// news feed updates. +[[ updateList ]] + +latestUpdatesTitle = Últimas actualizaciones + + // A listing of all Test Pilot experiments. [[ experimentsList ]] @@ -109,7 +115,8 @@ experimentListJustLaunchedTab = Recién lanzado experimentListJustUpdatedTab = Recién actualizado experimentListEndingTomorrow = Termina mañana experimentListEndingSoon = Termina pronto -experimentsListCondensedHeader = ¡Elegí tus experimentos! +experimentCondensedHeader = ¡Bienvenido a Test Pilot! +experimentListHeader = ¡Elegí tus experimentos! // An individual experiment in the listing of all Test Pilot experiments. @@ -280,3 +287,18 @@ noScriptLink = Encuentre el motivo viewPastExperiments = Ver experimentos pasados hidePastExperiments = Ocultar experimentos pasados + + +// Text of warnings to the user if various error conditions are detected +[[ warnings ]] + +warningGenericTitle = ¡Algo está mal! +warningGenericDetail = Algo falló con Test Pilot. Informá la falla y mencioná este mensaje de error. +warningUpgradeFirefoxTitle = ¡Actualizá Firefox para continuar! +warningUpgradeFirefoxDetail = Test Pilot requiere la última versión de Firefox. Actualizá Firefox para comenzar. +warningHttpsRequiredTitle = ¡HTTPS requerido! +warningHttpsRequiredDetail = Test Pilot debe accederse mediante HTTPS. Mirá nuestra documentación para más detalles. +warningMissingPrefTitle = ¿Desarrollando Test Pilot? +warningMissingPrefDetail = Al ejecutar Test Pilot localmente o en entornos de desarrollo, se necesita una configuración especial. Mirá nuestra documentación para más detalles. +warningBadHostnameTitle = ¡Nombre de servidor no aprobado! +warningBadHostnameDetail = El sitio de Test Pilot solamente puede ser accedido desde testpilot.firefox.com, testpilot.stage.mozaws.net, testpilot.dev.mozaws.net, o example.com:8000. Mirá nuestra documentación para más detalles. diff --git a/locales/es-ES/app.ftl b/locales/es-ES/app.ftl index ce18eeebb5..8b9f7eb3fd 100644 --- a/locales/es-ES/app.ftl +++ b/locales/es-ES/app.ftl @@ -284,3 +284,18 @@ noScriptLink = Descubre por qué viewPastExperiments = Ver experimentos antiguos hidePastExperiments = Ocultar experimentos antiguos + + +// Text of warnings to the user if various error conditions are detected +[[ warnings ]] + +warningGenericTitle = ¡Algo está mal! +warningGenericDetail = Algo ha fallado con Test Pilot. Por favor informa del error y menciona este mensaje de error. +warningUpgradeFirefoxTitle = ¡Actualiza Firefox para continuar! +warningUpgradeFirefoxDetail = Test Pilot requiere la última versión de Firefox. Actualiza Firefox para comenzar. +warningHttpsRequiredTitle = ¡Se requiere HTTPS! +warningHttpsRequiredDetail = Test Pilot debe ser accedido mediante HTTPS. Por favor mira nuestra documentación para más detalles. +warningMissingPrefTitle = ¿Desarrollando Test Pilot? +warningMissingPrefDetail = Al ejecutar Test Pilot localmente o en entornos de desarrollo, se necesita una configuración especial. Por favor mira nuestra documentación para más detalles. +warningBadHostnameTitle = ¡Nombre de servidor no aprobado! +warningBadHostnameDetail = El sitio de Test Pilot solo puede ser accedido desde testpilot.firefox.com, testpilot.stage.mozaws.net, testpilot.dev.mozaws.net, o example.com:8000. Mira nuestra documentación para más detalles. diff --git a/locales/fa/app.ftl b/locales/fa/app.ftl index 85a838a704..965d41471b 100644 --- a/locales/fa/app.ftl +++ b/locales/fa/app.ftl @@ -284,3 +284,17 @@ noScriptLink = ببینید چرا viewPastExperiments = نمایش آزمایش‌های گذشته hidePastExperiments = مخفی کردن آزمایش‌های گذشته + + +// Text of warnings to the user if various error conditions are detected +[[ warnings ]] + +warningGenericTitle = خطایی رخ داده است! +warningGenericDetail = مشکلی در خلبان آزمایشی رخ داده است. لطفا یک گزارش خطا ایجاد کنید و این پیغام خطا را ذکر نمایید. +warningUpgradeFirefoxTitle = برای ادامه، فایرفاکس را ارتقا دهید! +warningUpgradeFirefoxDetail = خلبان آزمایشی به آخرین نسخه فایرفاکس احتیاج دارد، برای شروع فایرفاکس را ارتقا دهید. +warningHttpsRequiredTitle = HTTPS مورد نیاز است! +warningHttpsRequiredDetail = خلبان آزمایشی باید از طریق HTTPS مورد استفاده قرار بگیرد. برای جزییات لطفا مستندات ما را ببینید. +warningMissingPrefDetail = وقتی خلبان آزمایشی را محلی یا در محیط‌های توسعه اجرا می‌کنید، پیکربندی ویژه‌ای مورد نیاز است. برای جزییات لطفا مستندات ما را ببینید. +warningBadHostnameTitle = نام میزبان تایید نشده! +warningBadHostnameDetail = ممکن است خلبان آزمایشی تنها از طریق testpilot.firefox.com، testpilot.stage.mozaws.net، testpilot.dev.mozaws.net یا example.com:8000 دردسترس باشد. برای جزییات لطفا مستندات ما را ببینید. diff --git a/locales/fy-NL/app.ftl b/locales/fy-NL/app.ftl index 10f1720c17..f8969f6565 100644 --- a/locales/fy-NL/app.ftl +++ b/locales/fy-NL/app.ftl @@ -284,3 +284,18 @@ noScriptLink = Besjoch wêrom viewPastExperiments = Alde eksperiminten besjen hidePastExperiments = Alde eksperiminten ferstopje + + +// Text of warnings to the user if various error conditions are detected +[[ warnings ]] + +warningGenericTitle = Der is wat mis! +warningGenericDetail = Der is wat misgien mei Test Pilot. Meld in bug en neam dizze flatermelding. +warningUpgradeFirefoxTitle = Fernij Firefox om troch te gean! +warningUpgradeFirefoxDetail = Test Pilot fereasket de nijste ferzje fan Firefox. Fernij Firefox om te begjinnen. +warningHttpsRequiredTitle = HTTPS fereaske! +warningHttpsRequiredDetail = Test Pilot moat fia HTTPS benadere wurde. Besjoch ús dokumintaasje foar details. +warningMissingPrefTitle = Untwikkelje jo Test Pilot? +warningMissingPrefDetail = Foar it lokaal of yn ûntwikkelomjouwingen útfieren fan Test Pilot is spesjale konfiguraasje fereaske. Besjoch ús dokumintaasje foar details. +warningBadHostnameTitle = Net-goedkarde hostnamme! +warningBadHostnameDetail = De Test Pilot-website mei allinnich fan testpilot.firefox.com, testpilot.stage.mozaws.net, testpilot.dev.mozaws.net of example.com:8000 ôf benadere wurde. Besjoch ús dokumintaasje foar details. diff --git a/locales/sl/app.ftl b/locales/sl/app.ftl index 98b32a8155..fe070d22b4 100644 --- a/locales/sl/app.ftl +++ b/locales/sl/app.ftl @@ -299,5 +299,6 @@ warningUpgradeFirefoxDetail = Test Pilot zahteva najnovejšo različico Firefoxa warningHttpsRequiredTitle = HTTPS zahtevan! warningHttpsRequiredDetail = Dostop do Test Pilota mora biti preko HTTPS. Za podrobnosti glejte našo dokumentacijo. warningMissingPrefTitle = Razvijate Test Pilot? +warningMissingPrefDetail = Kadar Test Pilot poganjate lokalno ali v razvojnih okoljih, je zahtevana posebna konfiguracija. Za podrobnosti glejte našo dokumentacijo. warningBadHostnameTitle = Neodobreno ime gostitelja! warningBadHostnameDetail = Do strani Test Pilota je omogočen dostop samo s testpilot.firefox.com, testpilot.stage.mozaws.net, testpilot.dev.mozaws.net ali example.com:8000. Za podrobnosti glejte našo dokumentacijo. diff --git a/locales/zh-TW/app.ftl b/locales/zh-TW/app.ftl index 5b321355a0..d66595cb2b 100644 --- a/locales/zh-TW/app.ftl +++ b/locales/zh-TW/app.ftl @@ -284,3 +284,14 @@ noScriptLink = 找出原因 viewPastExperiments = 檢視先前的實驗 hidePastExperiments = 隱藏先前的實驗 + + +// Text of warnings to the user if various error conditions are detected +[[ warnings ]] + +warningGenericTitle = 有些東西不對勁！ +warningUpgradeFirefoxTitle = 升級 Firefox 後再繼續使用！ +warningUpgradeFirefoxDetail = 需要使用最新版本的 Firefox 才能使用 Test Pilot。請升級 Firefox 開始使用。 +warningHttpsRequiredTitle = 必須使用 HTTPS！ +warningHttpsRequiredDetail = 必須透過 HTTPS 通訊協定存取 Test Pilot。請參考文件當中的詳細資訊。 +warningMissingPrefTitle = 正在開發 Test Pilot 嗎？