diff --git a/.github/ISSUE_TEMPLATE/release_checklist_template.md b/.github/ISSUE_TEMPLATE/release_checklist_template.md index 398bcbfc375..bd8f28f54ea 100644 --- a/.github/ISSUE_TEMPLATE/release_checklist_template.md +++ b/.github/ISSUE_TEMPLATE/release_checklist_template.md @@ -14,7 +14,7 @@ Code freeze for a milestone is the *second Friday of the sprint*, and most of re ## Feature complete [Monday, 1st week of next milestone] -- [ ] Create a branch for the *current* milestone and protect it through Settings on the repo (need admin privileges). After that master is tracking the next milestone. Usually done on the Monday after feature complete on Friday. +- [ ] Create a branch for the *current* milestone and protect it through Settings on the repo (need admin privileges). After that `main` is tracking the next milestone. Usually done on the Monday after feature complete on Friday. - [ ] Create an issue in the *upcoming* milestone: "What's New Entry for [release]" to track work for the SUMO page ([example](https://github.com/mozilla-mobile/focus-android/issues/1670)). - [ ] [Create an issue](https://github.com/mozilla-mobile/focus-android/issues/new?template=release_checklist_template.md&title=Releng+for+) in the *upcoming* milestone: "Releng for [release]" and copy this checklist into it. Assign the next assignee. - [ ] Go through the list of bugs closed during this sprint and make sure all they're all added to the correct milestone. @@ -22,7 +22,7 @@ Code freeze for a milestone is the *second Friday of the sprint*, and most of re - [ ] Add either `ReadyForQA` or `QANotNeeded` flags on each of the bugs in the current milestone. ## Final string Import [Wednesday, 1st week of next milestone] -- [ ] Cherry pick any new string import commits from `master` into the release branch +- [ ] Cherry pick any new string import commits from `main` into the release branch ## Beta Submission [Thursday, 1st week of next milestone] @@ -40,7 +40,7 @@ Code freeze for a milestone is the *second Friday of the sprint*, and most of re ## During Beta - [ ] Check Google Play for new crashes. File issues and triage. -- [ ] If bugs are considered release blocker then fix them on master and the milestone branch (cherry-pick / uplift) +- [ ] If bugs are considered release blocker then fix them on `main` and the milestone branch (cherry-pick / uplift) - [ ] If needed tag a new RC version (e.g. v1.0-RC2) and follow the submission checklist again. ## Release diff --git a/.taskcluster.yml b/.taskcluster.yml index 138b5296a5b..b01b1ed6771 100644 --- a/.taskcluster.yml +++ b/.taskcluster.yml @@ -33,7 +33,6 @@ tasks: && git config advice.detachedHead false && git checkout ${event.pull_request.head.sha} && echo "--" > .adjust_token - && python tools/l10n/check_locales.py && ./gradlew --no-daemon clean assembleFocusDebug assembleKlarNightly assembleRelease detekt ktlint lintFocusDebug lintKlarNightly assembleFocusDebugAndroidTest testFocusDebugUnitTest testKlarNightlyUnitTest && pip install "compare-locales>=5.0.2,<6.0" && compare-locales --validate l10n.toml . @@ -99,7 +98,7 @@ tasks: && git submodule update --init && git config advice.detachedHead false && git checkout ${event.after} - && python tools/taskcluster/schedule-master-build.py + && python tools/taskcluster/schedule-main-build.py artifacts: public: type: directory @@ -107,7 +106,7 @@ tasks: expires: {$fromNow: '1 week'} metadata: name: (Focus for Android) Schedule tasks - description: Scheduling tasks for master push + description: Scheduling tasks for main branch push owner: ${event.pusher.name}@users.noreply.github.com source: ${event.repository.url} ############################################################################### @@ -236,7 +235,7 @@ tasks: - -cx - >- git fetch origin - && git reset --hard origin/master + && git reset --hard origin/main && git submodule update --init && python tools/taskcluster/release.py \ --channel nightly \ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 34ac02b87e4..597b7bc1d7a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,4 +1,4 @@ # Contributing to Focus for Android Please see our guidelines in our shared-docs repo: -https://github.com/mozilla-mobile/shared-docs/blob/master/android/CONTRIBUTING.md +https://github.com/mozilla-mobile/shared-docs/blob/main/android/CONTRIBUTING.md diff --git a/README.md b/README.md index c6f221a216c..97782f34ce9 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,5 @@ # Firefox Focus for Android -[![Build Status](https://travis-ci.org/mozilla-mobile/focus-android.svg?branch=master)](https://travis-ci.org/mozilla-mobile/focus-android) -[![Task Status](https://github.taskcluster.net/v1/repository/mozilla-mobile/focus-android/master/badge.svg)](https://github.taskcluster.net/v1/repository/mozilla-mobile/focus-android/master/latest) -[![codecov](https://codecov.io/gh/mozilla-mobile/focus-android/branch/master/graph/badge.svg)](https://codecov.io/gh/mozilla-mobile/focus-android/branch/master) - - _Browse like no one’s watching. The new Firefox Focus automatically blocks a wide range of online trackers — from the moment you launch it to the second you leave it. Easily erase your history, passwords and cookies, so you won’t get followed by things like unwanted ads._ Firefox Focus provides automatic ad blocking and tracking protection on an easy-to-use private browser. @@ -22,7 +17,7 @@ We encourage you to participate in this open source project. We love Pull Reques Before you attempt to make a contribution please read the [Community Participation Guidelines](https://www.mozilla.org/en-US/about/governance/policies/participation/). -* [Guide to Contributing](https://github.com/mozilla-mobile/shared-docs/blob/master/android/CONTRIBUTING.md) (**New contributors start here!**) +* [Guide to Contributing](https://github.com/mozilla-mobile/shared-docs/blob/main/android/CONTRIBUTING.md) (**New contributors start here!**) * [View current Issues](https://github.com/mozilla-mobile/focus-android/issues), [view current Pull Requests](https://github.com/mozilla-mobile/focus-android/pulls), or [file a security issue][sec issue]. diff --git a/app/proguard-rules.pro b/app/proguard-rules.pro index 9da0aefb2ec..c5e43871317 100644 --- a/app/proguard-rules.pro +++ b/app/proguard-rules.pro @@ -15,33 +15,6 @@ # For more details, see # http://developer.android.com/guide/developing/tools/proguard.html -# Add any project specific keep options here: - -# The Buddybuild SDK builds using ACRA 4.6.4, which is afflictted -# by the following bug: https://github.com/ACRA/acra/issues/301 -# That is fixed in ACRA 4.7, but we need to wait for Buddybuild to upgrade -# for that to be fixed. (Note: this only affects BuddyBuild builds where -# the BuddyBuild SDK is enabled, and is not visible in local builds. You can -# enable the BuddyBuild SDK per branch for testing, by default it is only -# used for master.) --dontwarn org.acra.ErrorReporter - -# If your project uses WebView with JS, uncomment the following -# and specify the fully qualified class name to the JavaScript interface -# class: -#-keepclassmembers class fqcn.of.javascript.interface.for.webview { -# public *; -#} - -# Uncomment this to preserve the line number information for -# debugging stack traces. -#-keepattributes SourceFile,LineNumberTable - -# If you keep the line number information, uncomment this to -# hide the original source file name. -#-renamesourcefileattribute SourceFile - - #################################################################################################### # Adjust #################################################################################################### diff --git a/app/src/main/java/org/mozilla/focus/web/Download.java b/app/src/main/java/org/mozilla/focus/web/Download.java deleted file mode 100644 index 9a89ae46d5d..00000000000 --- a/app/src/main/java/org/mozilla/focus/web/Download.java +++ /dev/null @@ -1,97 +0,0 @@ -/* -*- Mode: Java; c-basic-offset: 4; tab-width: 4; indent-tabs-mode: nil; -*- - * This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ - -package org.mozilla.focus.web; - -import android.os.Parcel; -import android.os.Parcelable; - -public class Download implements Parcelable { - public static final Parcelable.Creator CREATOR = new Parcelable.Creator() { - - @Override - public Download createFromParcel(Parcel source) { - return new Download( - source.readString(), - source.readString(), - source.readString(), - source.readString(), - source.readLong(), - source.readString(), - source.readString()); - } - - @Override - public Download[] newArray(int size) { - return new Download[size]; - } - }; - - private final String url; - private final String contentDisposition; - private final String mimeType; - private final long contentLength; - private final String userAgent; - private final String destinationDirectory; - private final String fileName; - - public Download(String url, String userAgent, String contentDisposition, String mimeType, long contentLength, - String destinationDirectory, String fileName) { - this.url = url; - this.userAgent = userAgent; - this.contentDisposition = contentDisposition; - this.mimeType = mimeType; - this.contentLength = contentLength; - this.destinationDirectory = destinationDirectory; - this.fileName = fileName; - } - - /** - * @return a Environment.DIRECTORY_* constant. - */ - public String getDestinationDirectory() { - return destinationDirectory; - } - - public String getUrl() { - return url; - } - - public String getContentDisposition() { - return contentDisposition; - } - - public String getMimeType() { - return mimeType; - } - - public long getContentLength() { - return contentLength; - } - - public String getUserAgent() { - return userAgent; - } - - public String getFileName() { - return fileName; - } - - @Override - public int describeContents() { - return 0; - } - - @Override - public void writeToParcel(Parcel dest, int flags) { - dest.writeString(url); - dest.writeString(userAgent); - dest.writeString(contentDisposition); - dest.writeString(mimeType); - dest.writeLong(contentLength); - dest.writeString(destinationDirectory); - dest.writeString(fileName); - } -} diff --git a/app/src/main/java/org/mozilla/focus/web/HttpAuthenticationDialogBuilder.java b/app/src/main/java/org/mozilla/focus/web/HttpAuthenticationDialogBuilder.java deleted file mode 100644 index e1e9623ce60..00000000000 --- a/app/src/main/java/org/mozilla/focus/web/HttpAuthenticationDialogBuilder.java +++ /dev/null @@ -1,144 +0,0 @@ -/* This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ - -package org.mozilla.focus.web; - -import android.app.AlertDialog; -import android.content.Context; -import android.content.DialogInterface; -import androidx.core.content.ContextCompat; -import android.view.KeyEvent; -import android.view.LayoutInflater; -import android.view.View; -import android.view.WindowManager; -import android.view.inputmethod.EditorInfo; -import android.widget.TextView; -import android.widget.TextView.OnEditorActionListener; -import org.mozilla.focus.R; - -public class HttpAuthenticationDialogBuilder { - - private final Context context; - - private final String host; - private final String realm; - - private AlertDialog dialog; - private TextView usernameTextView; - private TextView passwordTextView; - - private OkListener okListener; - private CancelListener cancelListener; - - public static class Builder { - private final Context context; - private final String host; - private final String realm; - - private OkListener okListener; - private CancelListener cancelListener; - - public Builder(Context context, String host, String realm) { - this.context = context; - this.host = host; - this.realm = realm; - } - - public Builder setOkListener(OkListener okListener) { - this.okListener = okListener; - return this; - } - - public Builder setCancelListener(CancelListener cancelListener) { - this.cancelListener = cancelListener; - return this; - } - - public HttpAuthenticationDialogBuilder build() { - return new HttpAuthenticationDialogBuilder(this); - } - } - - public HttpAuthenticationDialogBuilder(Builder builder) { - context = builder.context; - host = builder.host; - realm = builder.realm; - okListener = builder.okListener; - cancelListener = builder.cancelListener; - } - - - private String getUsername() { - return usernameTextView.getText().toString(); - } - - private String getPassword() { - return passwordTextView.getText().toString(); - } - - public void show() { - dialog.show(); - int photonBlue = ContextCompat.getColor(context, R.color.photonBlue50); - dialog.getButton(DialogInterface.BUTTON_POSITIVE).setTextColor(photonBlue); - dialog.getButton(DialogInterface.BUTTON_NEGATIVE).setTextColor(photonBlue); - usernameTextView.requestFocus(); - } - - - public void createDialog() { - LayoutInflater inflater = LayoutInflater.from(context); - View view = inflater.inflate(R.layout.dialog_http_auth, null); - usernameTextView = view.findViewById(R.id.httpAuthUsername); - passwordTextView = view.findViewById(R.id.httpAuthPassword); - passwordTextView.setOnEditorActionListener(new OnEditorActionListener() { - @Override - public boolean onEditorAction(TextView v, int actionId, KeyEvent event) { - if (actionId == EditorInfo.IME_ACTION_DONE) { - dialog.getButton(AlertDialog.BUTTON_POSITIVE).performClick(); - return true; - } - return false; - } - }); - - buildDialog(view); - } - - private void buildDialog(View view) { - dialog = new AlertDialog.Builder(context) - .setIconAttribute(android.R.attr.alertDialogIcon) - .setView(view) - .setPositiveButton(R.string.action_ok, new DialogInterface.OnClickListener() { - public void onClick(DialogInterface dialog, int whichButton) { - if (okListener != null) { - okListener.onOk(host, realm, getUsername(), getPassword()); - } - } - }) - .setNegativeButton(R.string.action_cancel, new DialogInterface.OnClickListener() { - public void onClick(DialogInterface dialog, int whichButton) { - if (cancelListener != null) { - cancelListener.onCancel(); - } - } - }) - .setOnCancelListener(new DialogInterface.OnCancelListener() { - public void onCancel(DialogInterface dialog) { - if (cancelListener != null) { - cancelListener.onCancel(); - } - } - }) - .create(); - dialog.getWindow().setSoftInputMode(WindowManager.LayoutParams.SOFT_INPUT_STATE_VISIBLE); - } - - public interface OkListener { - void onOk(String host, String realm, String username, String password); - } - - public interface CancelListener { - void onCancel(); - } -} diff --git a/app/src/main/java/org/mozilla/focus/web/IFindListener.java b/app/src/main/java/org/mozilla/focus/web/IFindListener.java deleted file mode 100644 index fb31fb3ad39..00000000000 --- a/app/src/main/java/org/mozilla/focus/web/IFindListener.java +++ /dev/null @@ -1,5 +0,0 @@ -package org.mozilla.focus.web; - -import android.webkit.WebView; - -public interface IFindListener extends WebView.FindListener {} diff --git a/app/src/main/java/org/mozilla/focus/webview/ErrorPage.java b/app/src/main/java/org/mozilla/focus/webview/ErrorPage.java deleted file mode 100644 index 769fab6b7b1..00000000000 --- a/app/src/main/java/org/mozilla/focus/webview/ErrorPage.java +++ /dev/null @@ -1,128 +0,0 @@ -/* -*- Mode: Java; c-basic-offset: 4; tab-width: 20; indent-tabs-mode: nil; -*- - * This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ -package org.mozilla.focus.webview; - -import android.content.res.Resources; -import androidx.collection.ArrayMap; -import androidx.core.util.Pair; -import android.webkit.WebView; -import android.webkit.WebViewClient; - -import org.mozilla.focus.R; -import org.mozilla.focus.utils.HtmlLoader; - -import java.util.HashMap; -import java.util.Map; - -public class ErrorPage { - - private static final HashMap> errorDescriptionMap; - - static { - errorDescriptionMap = new HashMap<>(); - - // Chromium's mapping (internal error code, to Android WebView error code) is described at: - // https://chromium.googlesource.com/chromium/src.git/+/master/android_webview/java/src/org/chromium/android_webview/ErrorCodeConversionHelper.java - - errorDescriptionMap.put(WebViewClient.ERROR_UNKNOWN, - new Pair<>(R.string.error_connectionfailure_title, R.string.error_connectionfailure_message)); - - // This is probably the most commonly shown error. If there's no network, we inevitably - // show this. - errorDescriptionMap.put(WebViewClient.ERROR_HOST_LOOKUP, - new Pair<>(R.string.error_hostLookup_title, R.string.error_hostLookup_message)); - -// WebViewClient.ERROR_UNSUPPORTED_AUTH_SCHEME - // TODO: we don't actually handle this in firefox - does this happen in real life? - -// WebViewClient.ERROR_AUTHENTICATION - // TODO: there's no point in implementing this until we actually support http auth (#159) - - errorDescriptionMap.put(WebViewClient.ERROR_CONNECT, - new Pair<>(R.string.error_connect_title, R.string.error_connect_message)); - - // It's unclear what this actually means - it's not well documented. Based on looking at - // ErrorCodeConversionHelper this could happen if networking is disabled during load, in which - // case the generic error is good enough: - errorDescriptionMap.put(WebViewClient.ERROR_IO, - new Pair<>(R.string.error_connectionfailure_title, R.string.error_connectionfailure_message)); - - errorDescriptionMap.put(WebViewClient.ERROR_TIMEOUT, - new Pair<>(R.string.error_timeout_title, R.string.error_timeout_message)); - - errorDescriptionMap.put(WebViewClient.ERROR_REDIRECT_LOOP, - new Pair<>(R.string.error_redirectLoop_title, R.string.error_redirectLoop_message)); - - // We already try to handle external URLs if possible (i.e. we offer to open the corresponding - // app, if available for a given scheme). If we end up here that means no app exists. - // We could consider showing an "open google play" link here, but ultimately it's hard - // to know whether that's the right step, especially if there are no good apps for actually - // handling such a protocol there - moreover there doesn't seem to be a good way to search - // google play for apps supporting a given scheme. - errorDescriptionMap.put(WebViewClient.ERROR_UNSUPPORTED_SCHEME, - new Pair<>(R.string.error_unsupportedprotocol_title, R.string.error_unsupportedprotocol_message)); - - errorDescriptionMap.put(WebViewClient.ERROR_FAILED_SSL_HANDSHAKE, - new Pair<>(R.string.error_sslhandshake_title, R.string.error_sslhandshake_message)); - - errorDescriptionMap.put(WebViewClient.ERROR_BAD_URL, - new Pair<>(R.string.error_malformedURI_title, R.string.error_malformedURI_message)); - - // WebView returns ERROR_UNKNOWN when we try to access a file:/// on Android (with the error string - // containing access denied), so I'm not too sure why these codes exist: - // sure why these error codes exit -// WebViewClient.ERROR_FILE; -// WebViewClient.ERROR_FILE_NOT_FOUND; - - // Seems to be an indication of OOM, insufficient resources, or too many queued DNS queries - errorDescriptionMap.put(WebViewClient.ERROR_TOO_MANY_REQUESTS, - new Pair<>(R.string.error_generic_title, R.string.error_generic_message)); - } - - public static boolean supportsErrorCode(final int errorCode) { - return (errorDescriptionMap.get(errorCode) != null); - } - - public static void loadErrorPage(final WebView webView, final String desiredURL, final int errorCode) { - final Pair errorResourceIDs = errorDescriptionMap.get(errorCode); - - if (errorResourceIDs == null) { - throw new IllegalArgumentException("Cannot load error description for unsupported errorcode=" + errorCode); - } - - // This is quite hacky: ideally we'd just load the css file directly using a ' substitutionMap = new ArrayMap<>(); - - final Resources resources = webView.getContext().getResources(); - - substitutionMap.put("%page-title%", resources.getString(R.string.errorpage_title)); - substitutionMap.put("%button%", resources.getString(R.string.errorpage_refresh)); - - substitutionMap.put("%messageShort%", resources.getString(errorResourceIDs.first)); - substitutionMap.put("%messageLong%", resources.getString(errorResourceIDs.second, desiredURL)); - - substitutionMap.put("%css%", cssString); - - final String errorPage = HtmlLoader.loadResourceFile(webView.getContext(), R.raw.errorpage, substitutionMap); - - // We could load the raw html file directly into the webview using a file:///android_res/ - // URI - however we'd then need to do some JS hacking to do our String substitutions. Moreover - // we'd have to deal with the mixed-content issues detailed above in that case. - webView.loadDataWithBaseURL(desiredURL, errorPage, "text/html", "UTF8", desiredURL); - } -} diff --git a/app/src/main/java/org/mozilla/focus/webview/NestedWebView.java b/app/src/main/java/org/mozilla/focus/webview/NestedWebView.java deleted file mode 100644 index 0ed5c2c335b..00000000000 --- a/app/src/main/java/org/mozilla/focus/webview/NestedWebView.java +++ /dev/null @@ -1,140 +0,0 @@ -/* -*- Mode: Java; c-basic-offset: 4; tab-width: 20; indent-tabs-mode: nil; -*- - * This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ - -package org.mozilla.focus.webview; - -import android.content.Context; -import androidx.core.view.NestedScrollingChild; -import androidx.core.view.NestedScrollingChildHelper; -import androidx.core.view.ViewCompat; -import android.util.AttributeSet; -import android.view.MotionEvent; -import android.webkit.WebView; - -/** - * WebView that supports nested scrolls (for using in a CoordinatorLayout). - * - * This code is a simplified version of the NestedScrollView implementation - * which can be found in the support library: - * {@link androidx.core.widget.NestedScrollView} - * - * Based on: - * https://github.com/takahirom/webview-in-coordinatorlayout - */ -public class NestedWebView extends WebView implements NestedScrollingChild { - private int mLastY; - private final int[] mScrollOffset = new int[2]; - private final int[] mScrollConsumed = new int[2]; - private int mNestedOffsetY; - private final NestedScrollingChildHelper mChildHelper; - - public NestedWebView(Context context, AttributeSet attrs) { - super(context, attrs); - - mChildHelper = new NestedScrollingChildHelper(this); - setNestedScrollingEnabled(true); - } - - @Override - public boolean onTouchEvent(MotionEvent ev) { - final MotionEvent event = MotionEvent.obtain(ev); - final int action = ev.getActionMasked(); - - if (action == MotionEvent.ACTION_DOWN) { - mNestedOffsetY = 0; - } - - final int eventY = (int) event.getY(); - event.offsetLocation(0, mNestedOffsetY); - - switch (action) { - case MotionEvent.ACTION_MOVE: - int deltaY = mLastY - eventY; - - if (dispatchNestedPreScroll(0, deltaY, mScrollConsumed, mScrollOffset)) { - deltaY -= mScrollConsumed[1]; - event.offsetLocation(0, -mScrollOffset[1]); - mNestedOffsetY += mScrollOffset[1]; - } - - mLastY = eventY - mScrollOffset[1]; - - if (dispatchNestedScroll(0, mScrollOffset[1], 0, deltaY, mScrollOffset)) { - mLastY -= mScrollOffset[1]; - event.offsetLocation(0, mScrollOffset[1]); - mNestedOffsetY += mScrollOffset[1]; - } - break; - - case MotionEvent.ACTION_DOWN: - mLastY = eventY; - startNestedScroll(ViewCompat.SCROLL_AXIS_VERTICAL); - break; - - case MotionEvent.ACTION_UP: - case MotionEvent.ACTION_CANCEL: - stopNestedScroll(); - break; - - default: - // We don't care about other touch events - } - - // Execute event handler from parent class in all cases - boolean eventHandled = super.onTouchEvent(event); - - // Recycle previously obtained event - event.recycle(); - - return eventHandled; - } - - // NestedScrollingChild - - @Override - public void setNestedScrollingEnabled(boolean enabled) { - mChildHelper.setNestedScrollingEnabled(enabled); - } - - @Override - public boolean isNestedScrollingEnabled() { - return mChildHelper.isNestedScrollingEnabled(); - } - - @Override - public boolean startNestedScroll(int axes) { - return mChildHelper.startNestedScroll(axes); - } - - @Override - public void stopNestedScroll() { - mChildHelper.stopNestedScroll(); - } - - @Override - public boolean hasNestedScrollingParent() { - return mChildHelper.hasNestedScrollingParent(); - } - - @Override - public boolean dispatchNestedScroll(int dxConsumed, int dyConsumed, int dxUnconsumed, int dyUnconsumed, int[] offsetInWindow) { - return mChildHelper.dispatchNestedScroll(dxConsumed, dyConsumed, dxUnconsumed, dyUnconsumed, offsetInWindow); - } - - @Override - public boolean dispatchNestedPreScroll(int dx, int dy, int[] consumed, int[] offsetInWindow) { - return mChildHelper.dispatchNestedPreScroll(dx, dy, consumed, offsetInWindow); - } - - @Override - public boolean dispatchNestedFling(float velocityX, float velocityY, boolean consumed) { - return mChildHelper.dispatchNestedFling(velocityX, velocityY, consumed); - } - - @Override - public boolean dispatchNestedPreFling(float velocityX, float velocityY) { - return mChildHelper.dispatchNestedPreFling(velocityX, velocityY); - } -} diff --git a/app/src/main/java/org/mozilla/focus/webview/TelemetryAutofillCallback.kt b/app/src/main/java/org/mozilla/focus/webview/TelemetryAutofillCallback.kt deleted file mode 100644 index 6b5cff87188..00000000000 --- a/app/src/main/java/org/mozilla/focus/webview/TelemetryAutofillCallback.kt +++ /dev/null @@ -1,36 +0,0 @@ -/* This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ - -package org.mozilla.focus.webview - -import android.annotation.TargetApi -import android.content.Context -import android.os.Build -import android.view.View -import android.view.autofill.AutofillManager -import org.mozilla.focus.telemetry.TelemetryWrapper - -/** - * Callback implementation to send a telemetry event whenever an autocomplete input is shown. - */ -@TargetApi(Build.VERSION_CODES.O) -object TelemetryAutofillCallback : AutofillManager.AutofillCallback() { - fun register(context: Context) { - context.getSystemService(AutofillManager::class.java) - .registerCallback(TelemetryAutofillCallback) - } - - fun unregister(context: Context) { - context.getSystemService(AutofillManager::class.java) - .unregisterCallback(TelemetryAutofillCallback) - } - - override fun onAutofillEvent(view: View, virtualId: Int, event: Int) { - super.onAutofillEvent(view, virtualId, event) - - if (event == EVENT_INPUT_SHOWN) { - TelemetryWrapper.autofillShownEvent() - } - } -} diff --git a/app/src/main/java/org/mozilla/focus/webview/matcher/BlocklistProcessor.java b/app/src/main/java/org/mozilla/focus/webview/matcher/BlocklistProcessor.java deleted file mode 100644 index 7c1a0dbc541..00000000000 --- a/app/src/main/java/org/mozilla/focus/webview/matcher/BlocklistProcessor.java +++ /dev/null @@ -1,204 +0,0 @@ -/* -*- Mode: Java; c-basic-offset: 4; tab-width: 20; indent-tabs-mode: nil; -*- - * This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ -package org.mozilla.focus.webview.matcher; - -import android.util.JsonReader; -import android.util.JsonToken; - -import org.mozilla.focus.webview.matcher.util.FocusString; - -import java.io.IOException; -import java.util.Collections; -import java.util.HashSet; -import java.util.LinkedList; -import java.util.List; -import java.util.Map; -import java.util.Set; - -public class BlocklistProcessor { - - final static String SOCIAL = "Social"; - final static String DISCONNECT = "Disconnect"; - - private static final Set IGNORED_CATEGORIES; - - static { - final Set ignored = new HashSet<>(); - - ignored.add("Legacy Disconnect"); - ignored.add("Legacy Content"); - - IGNORED_CATEGORIES = Collections.unmodifiableSet(ignored); - } - - /** - * The sites in the "Disconnect" list that should be moved into "Social" - */ - private static final Set DISCONNECT_MOVED; - - static { - final Set moved = new HashSet<>(); - - moved.add("Facebook"); - moved.add("Twitter"); - - DISCONNECT_MOVED = Collections.unmodifiableSet(moved); - } - - public enum ListType { - BASE_LIST, - OVERRIDE_LIST - } - - public static Map loadCategoryMap(final JsonReader reader, final Map categoryMap, final ListType listType) throws IOException { - reader.beginObject(); - - while (reader.hasNext()) { - final String name = reader.nextName(); - - if (name.equals("categories")) { - extractCategories(reader, categoryMap, listType); - } else { - reader.skipValue(); - } - } - - reader.endObject(); - - return categoryMap; - } - - private interface UrlListCallback { - void put(final String url, final String siteOwner); - } - - private static class ListCallback implements UrlListCallback { - final List list; - final Set desiredOwners; - - /** - * @param desiredOwners A set containing all the site owners that should be stored in the list. - * Corresponds to the group owners listed in blocklist.json (e.g. "Facebook", "Twitter", etc.) - */ - ListCallback(final List list, final Set desiredOwners) { - this.list = list; - this.desiredOwners = desiredOwners; - } - - @Override - public void put(final String url, final String siteOwner) { - if (desiredOwners.contains(siteOwner)) { - list.add(url); - } - } - } - - private static class TrieCallback implements UrlListCallback { - final Trie trie; - - TrieCallback(final Trie trie) { - this.trie = trie; - } - - @Override - public void put(final String url, final String siteOwner) { - trie.put(FocusString.create(url).reverse()); - } - } - - private static void extractCategories(final JsonReader reader, final Map categoryMap, final ListType listType) throws IOException { - reader.beginObject(); - - final List socialOverrides = new LinkedList<>(); - - while (reader.hasNext()) { - final String categoryName = reader.nextName(); - - if (IGNORED_CATEGORIES.contains(categoryName)) { - reader.skipValue(); - } else if (categoryName.equals(DISCONNECT)) { - // We move these items into a different list, see below - ListCallback callback = new ListCallback(socialOverrides, DISCONNECT_MOVED); - extractCategory(reader, callback); - } else { - final Trie categoryTrie; - - if (listType == ListType.BASE_LIST) { - if (categoryMap.containsKey(categoryName)) { - throw new IllegalStateException("Cannot insert already loaded category"); - } - - categoryTrie = Trie.createRootNode(); - categoryMap.put(categoryName, categoryTrie); - } else { - categoryTrie = categoryMap.get(categoryName); - - if (categoryTrie == null) { - throw new IllegalStateException("Cannot add override items to nonexistent category"); - } - } - - final TrieCallback callback = new TrieCallback(categoryTrie); - - extractCategory(reader, callback); - } - } - - final Trie socialTrie = categoryMap.get(SOCIAL); - if (socialTrie == null && listType == ListType.BASE_LIST) { - throw new IllegalStateException("Expected social list to exist. Can't copy FB/Twitter into non-existing list"); - } - - for (final String url : socialOverrides) { - socialTrie.put(FocusString.create(url).reverse()); - } - - reader.endObject(); - } - - private static void extractCategory(final JsonReader reader, final UrlListCallback callback) throws IOException { - reader.beginArray(); - - while (reader.hasNext()) { - extractSite(reader, callback); - } - - reader.endArray(); - } - - private static void extractSite(final JsonReader reader, final UrlListCallback callback) throws IOException { - reader.beginObject(); - - final String siteOwner = reader.nextName(); - { - reader.beginObject(); - - while (reader.hasNext()) { - // We can get the site name using reader.nextName() here: - reader.skipValue(); - - JsonToken nextToken = reader.peek(); - - if (nextToken.name().equals("STRING")) { - // Sometimes there's a "dnt" entry, with unspecified purpose. - reader.skipValue(); - } else { - reader.beginArray(); - - while (reader.hasNext()) { - final String blockURL = reader.nextString(); - callback.put(blockURL, siteOwner); - } - - reader.endArray(); - } - } - - reader.endObject(); - } - - reader.endObject(); - } -} diff --git a/app/src/main/java/org/mozilla/focus/webview/matcher/EntityList.java b/app/src/main/java/org/mozilla/focus/webview/matcher/EntityList.java deleted file mode 100644 index a7b6a1cf330..00000000000 --- a/app/src/main/java/org/mozilla/focus/webview/matcher/EntityList.java +++ /dev/null @@ -1,63 +0,0 @@ -/* -*- Mode: Java; c-basic-offset: 4; tab-width: 20; indent-tabs-mode: nil; -*- - * This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ -package org.mozilla.focus.webview.matcher; - - -import android.net.Uri; -import android.text.TextUtils; - -import org.mozilla.focus.utils.UrlUtils; -import org.mozilla.focus.webview.matcher.Trie.WhiteListTrie; -import org.mozilla.focus.webview.matcher.util.FocusString; - -/* package-private */ class EntityList { - - private final WhiteListTrie rootNode; - - public EntityList() { - rootNode = WhiteListTrie.createRootNode(); - } - - public void putWhiteList(final FocusString revhost, final Trie whitelist) { - rootNode.putWhiteList(revhost, whitelist); - } - - public boolean isWhiteListed(final Uri site, final Uri resource) { - if (TextUtils.isEmpty(site.getHost()) || - TextUtils.isEmpty(resource.getHost()) || - site.getScheme().equals("data")) { - return false; - } - - if (UrlUtils.isPermittedResourceProtocol(resource.getScheme()) && - UrlUtils.isSupportedProtocol(site.getScheme())) { - final FocusString revSitehost = FocusString.create(site.getHost()).reverse(); - final FocusString revResourcehost = FocusString.create(resource.getHost()).reverse(); - - return isWhiteListed(revSitehost, revResourcehost, rootNode); - } else { - // This might be some imaginary/custom protocol: theguardian.com loads - // things like "nielsenwebid://nuid/999" and/or sets an iFrame URL to that: - return false; - } - } - - private boolean isWhiteListed(final FocusString site, final FocusString resource, final Trie revHostTrie) { - final WhiteListTrie next = (WhiteListTrie) revHostTrie.children.get(site.charAt(0)); - - if (next == null) { - // No matches - return false; - } - - if (next.whitelist != null && - next.whitelist.findNode(resource) != null) { - return true; - } - - return site.length() != 1 && isWhiteListed(site.substring(1), resource, next); - - } -} diff --git a/app/src/main/java/org/mozilla/focus/webview/matcher/EntityListProcessor.java b/app/src/main/java/org/mozilla/focus/webview/matcher/EntityListProcessor.java deleted file mode 100644 index a434455e459..00000000000 --- a/app/src/main/java/org/mozilla/focus/webview/matcher/EntityListProcessor.java +++ /dev/null @@ -1,79 +0,0 @@ -/* -*- Mode: Java; c-basic-offset: 4; tab-width: 20; indent-tabs-mode: nil; -*- - * This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ -package org.mozilla.focus.webview.matcher; - - -import android.util.JsonReader; - -import org.mozilla.focus.webview.matcher.util.FocusString; - -import java.io.IOException; -import java.util.ArrayList; - -/** - * Parses an entitylist json file, and returns an EntityList representation thereof. - */ -/* package-private */ class EntityListProcessor { - - private final EntityList entityMap = new EntityList(); - - public static EntityList getEntityMapFromJSON(final JsonReader reader) throws IOException { - EntityListProcessor processor = new EntityListProcessor(reader); - - return processor.entityMap; - } - - private EntityListProcessor(final JsonReader reader) throws IOException { - reader.beginObject(); - - while (reader.hasNext()) { - // We can get the siteName using reader.nextName() here - reader.skipValue(); - - handleSite(reader); - } - - reader.endObject(); - } - - private void handleSite(final JsonReader reader) throws IOException { - reader.beginObject(); - - final Trie whitelist = Trie.createRootNode(); - final ArrayList propertyList = new ArrayList<>(); - - while (reader.hasNext()) { - final String itemName = reader.nextName(); - - if (itemName.equals("properties")) { - reader.beginArray(); - - while (reader.hasNext()) { - propertyList.add(reader.nextString()); - } - - reader.endArray(); - } else if (itemName.equals("resources")) { - reader.beginArray(); - - while (reader.hasNext()) { - final FocusString revhost = FocusString.create(reader.nextString()).reverse(); - - whitelist.put(revhost); - } - - reader.endArray(); - } - } - - for (final String property : propertyList) { - final FocusString revhost = FocusString.create(property).reverse(); - - entityMap.putWhiteList(revhost, whitelist); - } - - reader.endObject(); - } -} diff --git a/app/src/main/java/org/mozilla/focus/webview/matcher/Trie.java b/app/src/main/java/org/mozilla/focus/webview/matcher/Trie.java deleted file mode 100644 index 5ec4cc8f7df..00000000000 --- a/app/src/main/java/org/mozilla/focus/webview/matcher/Trie.java +++ /dev/null @@ -1,110 +0,0 @@ -/* -*- Mode: Java; c-basic-offset: 4; tab-width: 20; indent-tabs-mode: nil; -*- - * This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ -package org.mozilla.focus.webview.matcher; - -import android.util.SparseArray; - -import org.mozilla.focus.webview.matcher.util.FocusString; - -/* package-private */ class Trie { - - /** - * Trie that adds storage for a whitelist (itself another trie) on each node. - */ - public static class WhiteListTrie extends Trie { - Trie whitelist = null; - - private WhiteListTrie(final char character, final WhiteListTrie parent) { - super(character, parent); - } - - @Override - protected Trie createNode(final char character, final Trie parent) { - return new WhiteListTrie(character, (WhiteListTrie) parent); - } - - public static WhiteListTrie createRootNode() { - return new WhiteListTrie(Character.MIN_VALUE, null); - } - - /* Convenience method so that clients aren't forced to do their own casting. */ - public void putWhiteList(final FocusString string, final Trie whitelist) { - WhiteListTrie node = (WhiteListTrie) super.put(string); - - if (node.whitelist != null) { - throw new IllegalStateException("Whitelist already set for node " + string); - } - - node.whitelist = whitelist; - } - } - - public final SparseArray children = new SparseArray<>(); - public boolean terminator = false; - - public Trie findNode(final FocusString string) { - if (terminator) { - // Match achieved - and we're at a domain boundary. This is important, because - // we don't want to return on partial domain matches. (E.g. if the trie node is bar.com, - // and the search string is foo-bar.com, we shouldn't match. But foo.bar.com should match.) - if (string.length() == 0 || string.charAt(0) == '.') { - return this; - } - } else if (string.length() == 0) { - // Finished the string, no match - return null; - } - - final Trie next = children.get(string.charAt(0)); - - if (next == null) { - return null; - } - - return next.findNode(string.substring(1)); - } - - public Trie put(final FocusString string) { - if (string.length() == 0) { - terminator = true; - return this; - } - - final char character = string.charAt(0); - - final Trie child = put(character); - - return child.put(string.substring(1)); - } - - public Trie put(char character) { - final Trie existingChild = children.get(character); - - if (existingChild != null) { - return existingChild; - } - - final Trie newChild = createNode(character, this); - - children.put(character, newChild); - - return newChild; - } - - private Trie(char character, Trie parent) { - if (parent != null) { - parent.children.put(character, this); - } - } - - public static Trie createRootNode() { - return new Trie(Character.MIN_VALUE, null); - } - - // Subclasses must override to provide their node implementation - protected Trie createNode(final char character, final Trie parent) { - return new Trie(character, parent); - } -} diff --git a/app/src/main/java/org/mozilla/focus/webview/matcher/UrlMatcher.java b/app/src/main/java/org/mozilla/focus/webview/matcher/UrlMatcher.java deleted file mode 100644 index 04222dcac5a..00000000000 --- a/app/src/main/java/org/mozilla/focus/webview/matcher/UrlMatcher.java +++ /dev/null @@ -1,283 +0,0 @@ -/* -*- Mode: Java; c-basic-offset: 4; tab-width: 20; indent-tabs-mode: nil; -*- - * This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ -package org.mozilla.focus.webview.matcher; - - -import android.content.Context; -import android.content.SharedPreferences; -import android.net.Uri; -import androidx.annotation.NonNull; -import androidx.annotation.Nullable; -import androidx.annotation.VisibleForTesting; -import androidx.collection.ArrayMap; -import androidx.preference.PreferenceManager; - -import android.util.JsonReader; - -import org.mozilla.focus.R; -import org.mozilla.focus.utils.Settings; -import org.mozilla.focus.webview.matcher.util.FocusString; - -import java.io.IOException; -import java.io.InputStreamReader; -import java.nio.charset.StandardCharsets; -import java.util.Collections; -import java.util.HashMap; -import java.util.HashSet; -import java.util.Map; -import java.util.Set; - -public class UrlMatcher implements SharedPreferences.OnSharedPreferenceChangeListener { - /** - * Map of pref to blocking category (preference key -> Blocklist category name). - */ - private final Map categoryPrefMap; - - private static final String[] WEBFONT_EXTENSIONS = new String[]{ - ".woff2", - ".woff", - ".eot", - ".ttf", - ".otf" - }; - - private static final String WEBFONTS = "Webfonts"; - - private static Map loadDefaultPrefMap(final Context context) { - Map tempMap = new ArrayMap<>(); - - tempMap.put(context.getString(R.string.pref_key_privacy_block_ads), "Advertising"); - tempMap.put(context.getString(R.string.pref_key_privacy_block_analytics), "Analytics"); - tempMap.put(context.getString(R.string.pref_key_privacy_block_social), "Social"); - tempMap.put(context.getString(R.string.pref_key_privacy_block_other), "Content"); - tempMap.put(context.getString(R.string.pref_key_privacy_block_cryptomining), "Cryptomining"); - tempMap.put(context.getString(R.string.pref_key_privacy_block_fingerprinting), "Fingerprinting"); - - // This is a "fake" category - webfont handling is independent of the blocklists - tempMap.put(context.getString(R.string.pref_key_performance_block_webfonts), WEBFONTS); - - return Collections.unmodifiableMap(tempMap); - } - - private final Map categories; - private final Set enabledCategories = new HashSet<>(); - - private final EntityList entityList; - // A cached list of previously matched URLs. This MUST be cleared whenever items are removed from enabledCategories. - private final HashSet previouslyMatched = new HashSet<>(); - // A cahced list of previously approved URLs. This MUST be cleared whenever items are added to enabledCategories. - private final HashSet previouslyUnmatched = new HashSet<>(); - - private boolean blockWebfonts = true; - - public static UrlMatcher loadMatcher(final Context context, final int blockListFile, final int[] blockListOverrides, final int entityListFile) { - final Map categoryPrefMap = loadDefaultPrefMap(context); - - final Map categoryMap = new HashMap<>(5); - try (final JsonReader jsonReader = - new JsonReader(new InputStreamReader(context.getResources().openRawResource(blockListFile), StandardCharsets.UTF_8))) { - BlocklistProcessor.loadCategoryMap(jsonReader, categoryMap, BlocklistProcessor.ListType.BASE_LIST); - } catch (IOException e) { - throw new IllegalStateException("Unable to parse blacklist"); - } - - if (blockListOverrides != null) { - for (int blockListOverride : blockListOverrides) { - try (final JsonReader jsonReader = - new JsonReader(new InputStreamReader(context.getResources().openRawResource(blockListOverride), StandardCharsets.UTF_8))) { - BlocklistProcessor.loadCategoryMap(jsonReader, categoryMap, BlocklistProcessor.ListType.OVERRIDE_LIST); - } catch (IOException e) { - throw new IllegalStateException("Unable to parse override blacklist"); - } - } - } - - final EntityList entityList; - try (final JsonReader jsonReader = new JsonReader(new InputStreamReader(context.getResources().openRawResource(entityListFile), StandardCharsets.UTF_8))) { - entityList = EntityListProcessor.getEntityMapFromJSON(jsonReader); - } catch (IOException e) { - throw new IllegalStateException("Unable to parse entity list"); - } - - return new UrlMatcher(context, categoryPrefMap, categoryMap, entityList); - } - - /* package-private */ UrlMatcher(final Context context, - @NonNull final Map categoryPrefMap, - @NonNull final Map categoryMap, - @Nullable final EntityList entityList) { - this.categoryPrefMap = categoryPrefMap; - this.entityList = entityList; - this.categories = categoryMap; - - // Ensure all categories have been declared, and enable by default (loadPrefs() will then - // enabled/disable categories that have actually been configured). - for (final Map.Entry entry: categoryMap.entrySet()) { - if (!categoryPrefMap.values().contains(entry.getKey())) { - throw new IllegalArgumentException("categoryMap contains undeclared category: " + entry.getKey()); - } - - // Failsafe: enable all categories (we load preferences in the next step anyway) - enabledCategories.add(entry.getKey()); - } - - loadPrefs(context); - - PreferenceManager.getDefaultSharedPreferences(context).registerOnSharedPreferenceChangeListener(this); - } - - @Override - public void onSharedPreferenceChanged(final SharedPreferences sharedPreferences, final String prefName) { - final String categoryName = categoryPrefMap.get(prefName); - - if (categoryName != null) { - final boolean prefValue = sharedPreferences.getBoolean(prefName, false); - - setCategoryEnabled(categoryName, prefValue); - } - } - - private void loadPrefs(final Context context) { - final SharedPreferences prefs = PreferenceManager.getDefaultSharedPreferences(context); - - for (final Map.Entry entry : categoryPrefMap.entrySet()) { - final boolean prefValue; - if (entry.getKey().equals(context.getString(R.string.pref_key_performance_block_webfonts))) { - prefValue = Settings.getInstance(context).shouldBlockWebFonts(); - } else if (entry.getKey().equals(context.getString(R.string.pref_key_privacy_block_social))) { - prefValue = Settings.getInstance(context).shouldBlockSocialTrackers(); - } else if (entry.getKey().equals(context.getString(R.string.pref_key_privacy_block_ads))) { - prefValue = Settings.getInstance(context).shouldBlockAdTrackers(); - } else if (entry.getKey().equals(context.getString(R.string.pref_key_privacy_block_analytics))) { - prefValue = Settings.getInstance(context).shouldBlockAnalyticTrackers(); - } else if (entry.getKey().equals(context.getString(R.string.pref_key_privacy_block_other))) { - prefValue = Settings.getInstance(context).shouldBlockOtherTrackers(); - } else { - prefValue = prefs.getBoolean(entry.getKey(), true); - } - setCategoryEnabled(entry.getValue(), prefValue); - } - } - - @VisibleForTesting UrlMatcher(final String[] patterns) { - final Map map = new HashMap<>(); - map.put("default", "default"); - categoryPrefMap = Collections.unmodifiableMap(map); - - categories = new HashMap<>(); - - buildMatcher(patterns); - - entityList = null; - } - - /** - * Only used for testing - uses a list of urls to populate a "default" category. - * @param patterns - */ - private void buildMatcher(String[] patterns) { - final Trie defaultCategory; - if (!categories.containsKey("default")) { - defaultCategory = Trie.createRootNode(); - categories.put("default", defaultCategory); - } else { - defaultCategory = categories.get("default"); - } - - for (final String pattern : patterns) { - defaultCategory.put(FocusString.create(pattern).reverse()); - } - - enabledCategories.add("default"); - } - - public Set getCategories() { - return categories.keySet(); - } - - public void setCategoryEnabled(final String category, final boolean enabled) { - if (WEBFONTS.equals(category)) { - blockWebfonts = enabled; - return; - } - - if (!getCategories().contains(category)) { - throw new IllegalArgumentException("Can't enable/disable inexistant category"); - } - - if (enabled) { - if (enabledCategories.contains(category)) { - // Early return - nothing to do if the category is already enabled - } else { - enabledCategories.add(category); - previouslyUnmatched.clear(); - } - } else { - if (!enabledCategories.contains(category)) { - // Early return - nothing to do if the category is already disabled - } else { - enabledCategories.remove(category); - previouslyMatched.clear(); - } - - } - } - - public boolean matches(final Uri resourceURI, final Uri pageURI) { - final String path = resourceURI.getPath(); - - if (path == null) { - return false; - } - - // We need to handle webfonts first: if they are blocked, then whitelists don't matter. - // If they aren't blocked we still need to check domain blacklists below. - if (blockWebfonts) { - for (final String extension : WEBFONT_EXTENSIONS) { - if (path.endsWith(extension)) { - return true; - } - } - } - - final String resourceURLString = resourceURI.toString(); - - // Cached whitelisted items can be permitted now (but blacklisted needs to wait for the override / entity list) - if (previouslyUnmatched.contains(resourceURLString)) { - return false; - } - - if (entityList != null && - entityList.isWhiteListed(pageURI, resourceURI)) { - // We must not cache entityList items (and/or if we did, we'd have to clear the cache - // on every single location change) - return false; - } - - final String resourceHost = resourceURI.getHost(); - final String pageHost = pageURI.getHost(); - - if (pageHost != null && pageHost.equals(resourceHost)) { - return false; - } - - if (previouslyMatched.contains(resourceURLString)) { - return true; - } - - final FocusString revhost = FocusString.create(resourceHost).reverse(); - - for (final Map.Entry category : categories.entrySet()) { - if (enabledCategories.contains(category.getKey()) && - category.getValue().findNode(revhost) != null) { - previouslyMatched.add(resourceURLString); - return true; - } - } - - previouslyUnmatched.add(resourceURLString); - return false; - } -} diff --git a/app/src/main/java/org/mozilla/focus/webview/matcher/util/FocusString.java b/app/src/main/java/org/mozilla/focus/webview/matcher/util/FocusString.java deleted file mode 100644 index 7921e52d871..00000000000 --- a/app/src/main/java/org/mozilla/focus/webview/matcher/util/FocusString.java +++ /dev/null @@ -1,106 +0,0 @@ -/* -*- Mode: Java; c-basic-offset: 4; tab-width: 20; indent-tabs-mode: nil; -*- - * This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ -package org.mozilla.focus.webview.matcher.util; - -import androidx.annotation.CheckResult; - -/** - * A String wrapper utility that allows for efficient string reversal. - * - * We regularly need to reverse strings. The standard way of doing this in Java would be to copy - * the string to reverse (e.g. using StringBuffer.reverse()). This seems wasteful when we only - * read our Strings character by character, in which case can just transpose positions as needed. - */ -public abstract class FocusString { - protected final String string; - - protected abstract boolean isReversed(); - - private FocusString(final String string, final int offsetStart, final int offsetEnd) { - this.string = string; - this.offsetStart = offsetStart; - this.offsetEnd = offsetEnd; - - if (offsetStart > offsetEnd || offsetStart < 0 || offsetEnd < 0) { - throw new StringIndexOutOfBoundsException("Cannot create negative-length String"); - } - } - - public static FocusString create(final String string) { - return new ForwardString(string, 0, string.length()); - } - - public int length() { - return offsetEnd - offsetStart; - } - - // offset at the start of the _raw_ input String - final int offsetStart; - // offset at the end of the _raw_ input String - final int offsetEnd; - - @CheckResult public FocusString reverse() { - if (isReversed()) { - return new ForwardString(string, offsetStart, offsetEnd); - } else { - return new ReverseString(string, offsetStart, offsetEnd); - } - } - - public abstract char charAt(final int position); - - public abstract FocusString substring(final int startIndex); - - private static class ForwardString extends FocusString { - public ForwardString(final String string, final int offsetStart, final int offsetEnd) { - super(string, offsetStart, offsetEnd); - } - - @Override - protected boolean isReversed() { - return false; - } - - @Override - public char charAt(int position) { - if (position > length()) { - throw new StringIndexOutOfBoundsException(); - } - - return string.charAt(position + offsetStart); - } - - @Override - public FocusString substring(final int startIndex) { - // Just a normal substring - return new ForwardString(string, offsetStart + startIndex, offsetEnd); - } - } - - private static class ReverseString extends FocusString { - public ReverseString(final String string, final int offsetStart, final int offsetEnd) { - super(string, offsetStart, offsetEnd); - } - - @Override - protected boolean isReversed() { - return true; - } - - @Override - public char charAt(int position) { - if (position > length()) { - throw new StringIndexOutOfBoundsException(); - } - - return string.charAt(length() - 1 - position + offsetStart); - } - - @Override - public FocusString substring(int startIndex) { - return new ReverseString(string, offsetStart, offsetEnd - startIndex); - } - } -} diff --git a/app/src/main/res/raw/errorpage.html b/app/src/main/res/raw/errorpage.html deleted file mode 100644 index fd1eabd5183..00000000000 --- a/app/src/main/res/raw/errorpage.html +++ /dev/null @@ -1,38 +0,0 @@ - - - - - - - - %pageTitle% - - - - - -
- - -
-

%messageShort%

-
- - -
- - -
-

%messageLong%

-
- -
- - - - -
- - diff --git a/app/src/main/res/raw/errorpage_style.css b/app/src/main/res/raw/errorpage_style.css deleted file mode 100644 index 1a5ce6001b7..00000000000 --- a/app/src/main/res/raw/errorpage_style.css +++ /dev/null @@ -1,225 +0,0 @@ -/* This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ - -html, -body { - margin: 0; - padding: 0; - height: 100%; - --moz-vertical-spacing: 10px; - --moz-background-height: 32px; -} - -body { - background-size: 64px var(--moz-background-height); - /* background-size: 64px 32px; */ - background-repeat: repeat-x; - - background-color: #363B40; - color: #FFFFFF; - padding: 0 20px; - - font-weight: 300; - font-size: 13px; - -moz-text-size-adjust: none; - font-family: sans-serif; -} - - -ul { - /* Shove the list indicator so that its left aligned, but use outside so that text - * doesn't don't wrap the text around it */ - padding: 0 1em; - margin: 0; - list-style: round outside none; -} - -#errorShortDesc, -li:not(:last-of-type) { - /* Margins between the li and buttons below it won't be collapsed. Remove the bottom margin here. */ - margin: var(--moz-vertical-spacing) 0; -} - -li > button { - /* Removing the normal padding on the li so this stretched edge to edge. */ - margin-left: -1em; - margin-right: -1em; - width: calc(100% + 2em); -} - -/* Push the #ignoreWarningButton to the bottom on the blocked site page */ -.blockedsite > #errorPageContainer > #errorLongContent { - flex: 1; -} - -h1 { - margin: 0; - /* Since this has an underline, use padding for vertical spacing rather than margin */ - padding: var(--moz-vertical-spacing) 0; - font-weight: 300; - border-bottom: 1px solid #e0e2e5; -} - -h2 { - font-size: small; - padding: 0; - margin: var(--moz-vertical-spacing) 0; -} - -p { - margin: var(--moz-vertical-spacing) 0; -} - -button { - /* Force buttons to display: block here to try and enfoce collapsing margins */ - display: block; - width: 100%; - border: none; - padding: 1rem; - font-family: sans-serif; - background-color: #00A4DC; - color: #FFFFFF; - font-weight: 300; - border-radius: 2px; - background-image: none; - margin: var(--moz-vertical-spacing) 0 0; -} - -button.inProgress { - background-image: linear-gradient(-45deg, #dfe8ee, #dfe8ee 33%, - #ecf0f3 33%, #ecf0f3 66%, - #dfe8ee 66%, #dfe8ee); - background-size: 37px 5px; - background-repeat: repeat-x; - animation: progress 6s linear infinite; -} - -@keyframes progress { - from { background-position: 0 100%; } - to { background-position: 100% 100%; } -} - -.certerror { - background-image: linear-gradient(-45deg, #f0d000, #f0d000 33%, - #fedc00 33%, #fedc00 66%, - #f0d000 66%, #f0d000); -} - -.blockedsite { - background-image: linear-gradient(-45deg, #9b2e2e, #9b2e2e 33%, - #a83232 33%, #a83232 66%, - #9b2e2e 66%, #9b2e2e); - background-color: #b14646; - color: white; -} - -#errorPageContainer { - /* If the page is greater than 550px center the content. - * This number should be kept in sync with the media query for tablets below */ - max-width: 550px; - margin: 0 auto; - transform: translateY(var(--moz-background-height)); - padding-bottom: var(--moz-vertical-spacing); - - min-height: calc(100% - var(--moz-background-height) - var(--moz-vertical-spacing)); - display: flex; - flex-direction: column; -} - -/* Expanders have a structure of - *
- *

Title

- *

Content

- *
- * - * This shows an arrow to the right of the h2 element, and hides the content when collapsed="true". */ -.expander { - margin: var(--moz-vertical-spacing) 0; - background-image: url("chrome://browser/skin/images/dropmarker.svg"); - background-repeat: no-repeat; - /* dropmarker.svg is 10x7. Ensure that its centered in the middle of an 18x18 box */ - background-position: 3px 5.5px; - background-size: 10px 7px; - padding-left: 18px; -} - -div[collapsed="true"] > .expander { - background-image: url("chrome://browser/skin/images/dropmarker-right.svg"); - /* dropmarker.svg is 7x10. Ensure that its centered in the middle of an 18x18 box */ - background-size: 7px 10px; - background-position: 5.5px 4px; -} - -div[hidden] > .expander, -div[hidden] > .expander + *, -div[collapsed="true"] > .expander + * { - display: none; -} - -.blockedsite h1 { - border-bottom-color: #9b2e2e; -} - -.blockedsite button { - background-color: #9b2e2e; - color: white; -} - -/* Style warning button to look like a small text link in the - bottom. This is preferable to just using a text link - since there is already a mechanism in browser.js for trapping - oncommand events from unprivileged chrome pages (ErrorPageEventHandler).*/ -#ignoreWarningButton { - width: calc(100% + 40px); - -moz-appearance: none; - background: #b14646; - border: none; - text-decoration: underline; - margin: 0; - margin-inline-start: -20px; - font-size: smaller; - border-radius: 0; -} - -/* On large screen devices (hopefully a 7+ inch tablet, we already center content (see #errorPageContainer above). - Apply tablet specific styles here */ -@media (min-width: 550px) { - button { - min-width: 160px; - width: auto; - } - - /* If the tablet is tall as well, add some padding to make content feel a bit more centered */ - @media (min-height: 550px) { - #errorPageContainer { - padding-top: 64px; - min-height: calc(100% - 64px); - } - } -} - -#searchbox { - padding: 0; - display: flex; - margin: var(--moz-vertical-spacing) -1em; -} - -#searchbox > input { - flex: 3; - padding: 0em 3em 0em 1em; - width: 100%; - border: none; - font-family: sans-serif; - background-image: none; - background-color: white; - border-radius-top-right: none; - border-radius-bottom-right: none; -} - -#searchbox > button { - flex: 1; - margin: 0; - width: auto; -} - diff --git a/app/src/main/res/raw/google_mapping.json b/app/src/main/res/raw/google_mapping.json deleted file mode 100644 index e09ce2fb3c4..00000000000 --- a/app/src/main/res/raw/google_mapping.json +++ /dev/null @@ -1,55 +0,0 @@ -{ - "categories": { - "Advertising": [ - { - "Google": { - "http://www.google.com/": [ - "2mdn.net", - "admeld.com", - "admob.com", - "cc-dt.com", - "destinationurl.com", - "doubleclick.net", - "adwords.google.com", - "googleadservices.com", - "googlesyndication.com", - "googletagservices.com", - "invitemedia.com", - "smtad.net", - "teracent.com", - "teracent.net", - "ytsa.net" - ] - } - } - ], - "Analytics": [ - { - "Google": { - "http://www.google.com/": [ - "google-analytics.com", - "postrank.com" - ] - } - } - ], - "Social": [ - { - "Google": { - "http://www.google.com/": [ - "developers.google.com", - "gmail.com", - "mail.google.com", - "inbox.google.com", - "orkut.com", - "plus.google.com", - "plusone.google.com", - "voice.google.com", - "wave.google.com", - "googlemail.com" - ] - } - } - ] - } -} diff --git a/app/src/test/java/org/mozilla/focus/web/DownloadTest.java b/app/src/test/java/org/mozilla/focus/web/DownloadTest.java deleted file mode 100644 index 19f8f399d66..00000000000 --- a/app/src/test/java/org/mozilla/focus/web/DownloadTest.java +++ /dev/null @@ -1,70 +0,0 @@ -/* -*- Mode: Java; c-basic-offset: 4; tab-width: 4; indent-tabs-mode: nil; -*- - * This Source Code Form is subject to the terms of the Mozilla Public - * License, v. 2.0. If a copy of the MPL was not distributed with this - * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ - -package org.mozilla.focus.web; - -import android.os.Environment; -import android.os.Parcel; - -import org.junit.Test; -import org.junit.runner.RunWith; -import org.robolectric.RobolectricTestRunner; - -import static org.junit.Assert.assertEquals; - -@RunWith(RobolectricTestRunner.class) -public class DownloadTest { - private static final String fileName = "filename.png"; - @Test - public void testGetters() { - final Download download = new Download( - "https://www.mozilla.org/image.png", - "Focus/1.0", - "Content-Disposition: attachment; filename=\"filename.png\"", - "image/png", - 1024, - Environment.DIRECTORY_DOWNLOADS, - fileName); - - assertEquals("https://www.mozilla.org/image.png", download.getUrl()); - assertEquals("Focus/1.0", download.getUserAgent()); - assertEquals("Content-Disposition: attachment; filename=\"filename.png\"", download.getContentDisposition()); - assertEquals("image/png", download.getMimeType()); - assertEquals(1024, download.getContentLength()); - assertEquals(Environment.DIRECTORY_DOWNLOADS, download.getDestinationDirectory()); - assertEquals(fileName, download.getFileName()); - } - - @Test - public void testParcelable() { - final Parcel parcel = Parcel.obtain(); - - { - final Download download = new Download( - "https://www.mozilla.org/image.png", - "Focus/1.0", - "Content-Disposition: attachment; filename=\"filename.png\"", - "image/png", - 1024, - Environment.DIRECTORY_PICTURES, - fileName); - download.writeToParcel(parcel, 0); - } - - parcel.setDataPosition(0); - - { - final Download download = Download.CREATOR.createFromParcel(parcel); - - assertEquals("https://www.mozilla.org/image.png", download.getUrl()); - assertEquals("Focus/1.0", download.getUserAgent()); - assertEquals("Content-Disposition: attachment; filename=\"filename.png\"", download.getContentDisposition()); - assertEquals("image/png", download.getMimeType()); - assertEquals(1024, download.getContentLength()); - assertEquals(Environment.DIRECTORY_PICTURES, download.getDestinationDirectory()); - assertEquals(fileName, download.getFileName()); - } - } -} \ No newline at end of file diff --git a/tools/l10n/android2po/README.md b/tools/l10n/android2po/README.md deleted file mode 100644 index ede8e30e6be..00000000000 --- a/tools/l10n/android2po/README.md +++ /dev/null @@ -1,4 +0,0 @@ -This is an imported and forked version of android2po with some modifications to import and export -files that Pontoon can handle. - -https://github.com/miracle2k/android2po diff --git a/tools/l10n/android2po/a2po.py b/tools/l10n/android2po/a2po.py deleted file mode 100644 index d47f0e6ddd9..00000000000 --- a/tools/l10n/android2po/a2po.py +++ /dev/null @@ -1,4 +0,0 @@ -from program import run - -if __name__ == "__main__": - run() diff --git a/tools/l10n/android2po/commands.py b/tools/l10n/android2po/commands.py deleted file mode 100644 index 82d00717768..00000000000 --- a/tools/l10n/android2po/commands.py +++ /dev/null @@ -1,609 +0,0 @@ -from __future__ import absolute_import - -import os -import collections - -try: - from cStringIO import StringIO as BytesIO -except ImportError: # pragma: no cover - from io import BytesIO -from lxml import etree -from babel.messages import pofile, Catalog -from termcolor import colored - -import convert -from patch import read_po -from env import resolve_locale - -__all__ = ('CommandError', 'ExportCommand', 'ImportCommand', 'InitCommand',) - - -class CommandError(Exception): - pass - - -def read_catalog(filename, **kwargs): - """Helper to read a catalog from a .po file. - """ - f = open(filename, 'r') - try: - return read_po(f, **kwargs) - finally: - f.close() - - -def catalog2string(catalog, **kwargs): - """Helper that returns a babel message catalog as a string. - - This is a simple shortcut around pofile.write_po(). - """ - sf = BytesIO() - pofile.write_po(sf, catalog, **kwargs) - return sf.getvalue().decode('utf-8') - - -def xml2string(tree, action): - """Helper that returns a ``ResourceTree`` as an XML string. - - TODO: It would be cool if this could try to recreate the formatting - of the original xml file. - """ - ENCODING = 'utf-8' - dom = convert.write_xml(tree, warnfunc=action.message) - return etree.tostring(dom, xml_declaration=True, - encoding=ENCODING, pretty_print=True).decode('utf-8') - - -def read_xml(action, filename, **kw): - """Wrapper around the base read_xml() that pipes warnings - into the given action. - - Also handles errors and returns false if the file is invalid. - """ - try: - return convert.read_xml(filename, warnfunc=action.message, **kw) - except convert.InvalidResourceError as e: - action.done('failed') - action.message('Failed parsing "%s": %s' % (filename.rel, e), 'error') - return False - - -def xml2po(env, action, *a, **kw): - """Wrapper around the base xml2po() that uses the filters configured - by the environment. - """ - - def xml_filter(name): - for filter in env.config.ignores: - if filter.match(name): - return True - - kw['resfilter'] = xml_filter - if action: - kw['warnfunc'] = action.message - return convert.xml2po(*a, **kw) - - -def po2xml(env, action, *a, **kw): - """Wrapper around the base po2xml() that uses the filters configured - by the environment. - """ - - def po_filter(message): - if env.config.ignore_fuzzy and message.fuzzy: - return True - - kw['resfilter'] = po_filter - kw['warnfunc'] = action.message - return convert.po2xml(*a, **kw) - - -def get_catalog_counts(catalog): - """Return 3-tuple (total count, number of translated strings, number - of fuzzy strings), based on the given gettext catalog. - """ - # Make sure we don't count the header - return (len(catalog), - len([m for m in catalog if m.string and m.id]), - len([m for m in catalog if m.string and m.id and m.fuzzy])) - - -def list_languages(source, env, writer): - """Return a list of languages (by simply calling the proper - environment method. - - However, commands should use this helper rather than working - with the environment directly, as this outputs helpful - diagnostic messages along the way. - """ - assert source in ('gettext', 'android') - languages = getattr( - env, - 'get_gettext_languages' if source == 'gettext' else 'get_android_languages')() - lstr = ", ".join(map(str, languages)) - writer.action('info', - "Found %d language(s): %s" % (len(languages), lstr)) - writer.message('List of languages was based on %s' % ( - 'the existing gettext catalogs' if source == 'gettext' - else 'the existing Android resource directories' - )) - return languages - - -def ensure_directories(cmd, path): - """Ensure that the given directory exists. - """ - # Collect all the individual directories we need to create. - # Yes, I know about os.makedirs(), but I'd like to print out - # every single directory created. - needs_creating = [] - while not path.exists(): - if path in needs_creating: - break - needs_creating.append(path) - path = path.dir - - for path in reversed(needs_creating): - cmd.w.action('mkdir', path) - os.mkdir(path) - - -def write_file(cmd, filename, content, update=True, action=None, - ignore_exists=False): - """Helper that writes a file, while sending the proper actions - to the command's writer for stdout display of what's going on. - - ``content`` may be a callable. This is useful if you would like - to exploit the ``update=False`` check this function provides, - rather than doing that yourself before bothering to generate the - content you want to write. - - When ``update`` is not set, then if the file already exists we don't - change or overwrite it. - - If a Writer.Action is given in ``action``, it will be used to print - out messages. Otherwise, a new action will be started using the - filename as the text. If ``action`` is ``False``, nothing will be - printed. - """ - if action is None: - action = cmd.w.begin(filename) - - if filename.exists(): - if not update: - if ignore_exists: - # Downgade level of this message - action.update(severity='info') - action.done('exists') - return False - else: - old_hash = filename.hash() - else: - old_hash = None - - ensure_directories(cmd, filename.dir) - - f = open(filename, 'wb') - try: - if isinstance(content, collections.Callable): - content = content() - f.write(content.encode('utf-8')) - f.flush() - finally: - f.close() - - if action is not False: - if old_hash is None: - action.done('created') - elif old_hash != filename.hash(): - action.done('updated') - else: - # Note that this is merely for user information. We - # nevertheless wrote a new version of the file, we can't - # actually determine a change without generating the new - # version. - action.done('unchanged') - return True - - -class Command(object): - """Abstract base command class. - """ - - def __init__(self, env, writer): - self.env = env - self.w = writer - - @classmethod - def setup_arg_parser(cls, argparser): - """A command should register it's sub-arguments here with the - given argparser instance. - """ - - def execute(self): - raise NotImplementedError() - - -class InitCommand(Command): - """The init command; to initialize new languages. - """ - - @classmethod - def setup_arg_parser(cls, parser): - parser.add_argument('language', nargs='*', - help='Language code to initialize. If none given, all ' + - 'languages lacking a .po file will be initialized.') - - def make_or_get_template(self, kind, read_action=None, do_write=False, - update=True): - """Return the .pot template file (as a Catalog) for the given kind. - - If ``do_write`` is given, the template file will be saved in the - proper location. If ``update`` is ``False``, then an existing file - will not be overridden, however. - - If ``do_write`` is disabled, then you need to given ``read_action``, - the action which needs the template. This is so we can fail the - proper action if generating the template goes wrong. - - Once generated, the template will be cached as a class member, - and on subsequent access the cached version is returned. - """ - # Implement caching - only generate the catalog the first time - # this function is called. - if not hasattr(self, '_template_catalogs'): - self._template_catalogs = {} - - if kind in self._template_catalogs: - return self._template_catalogs[kind], False - - # Only one, xor the other. - assert read_action or do_write and not (read_action and do_write) - - template_pot = self.env.default.po(kind) - if do_write: - action = self.w.begin(template_pot) - else: - action = read_action - - # Read the XML, bail out if that fails - xmldata = read_xml(action, self.env.default.xml(kind)) - if xmldata is False: - return False, False - - # Actually generate the catalog - template_catalog = xml2po(self.env, action, xmldata) - self._template_catalogs[kind] = template_catalog - - # Write the catalog as a template to disk if necessary. - something_written = False - if do_write: - # Note that this is always rendered with "ignore_exists", - # i.e. we only log this action if we change the template. - if write_file(self, template_pot, - content=lambda: catalog2string(template_catalog), - action=action, ignore_exists=True, update=update): - something_written = True - - return template_catalog, something_written - - def generate_templates(self, update=True): - """Generate the template files. - - Do this only if they are not disabled. - """ - something_written = False - if not self.env.config.no_template: - for kind in self.env.xmlfiles: - _, write_happend = self.make_or_get_template( - kind, do_write=True, update=update) - if write_happend: - something_written = True - return something_written - - def generate_po(self, target_po_file, default_data, action, - language_data=None, language_data_files=None, - update=True, ignore_exists=False): - """Helper to generate a .po file. - - ``default_data`` is the collective data from the language neutral XML - files, and this is what the .po we generate will be based on. - - ``language_data`` is collective data from the corresponding - language-specific XML files, in case such data is available. - - ``language_data_files`` is the list of files that ``language_data`` - is based upon. This is because in some cases multiple XML files - might need to be combined into one gettext catalog. - - If ``update`` is not set than we will bail out early - if the file doesn't exist. - """ - - # This is a function so that it only will be run if write_file() - # actually needs it. - def make_catalog(): - if language_data is not None: - action.message('Using existing translations from %s' % ", ".join( - [l.rel for l in language_data_files])) - lang_catalog, unmatched = xml2po(self.env, action, - default_data, - language_data) - if unmatched: - action.message("Existing translation XML files for this " - "language contains strings not found in the " - "default XML files: %s" % (", ".join(unmatched))) - else: - action.message('No corresponding XML exists, generating catalog ' + - 'without translations') - lang_catalog = xml2po(self.env, action, default_data) - - catalog = catalog2string(lang_catalog) - - num_total, num_translated, _ = get_catalog_counts(lang_catalog) - action.message("%d strings processed, %d translated." % ( - num_total, num_translated)) - return catalog - - return write_file(self, target_po_file, content=make_catalog, - action=action, update=update, - ignore_exists=ignore_exists) - - def _iterate(self, language, require_translation=True): - """Yield 5-tuples in the form of: ( - action object, - target .po file, - source xml data, - translated xml data, - list of files translated xml data was read from - ) - - This is implemeted as a separate iterator so that later on we can - also support a mechanism in which multiple xml files are stored in - one .po file, i.e. on export, multiple xml files needs to be able - to yield into a single .po target. - """ - for kind in self.env.xmlfiles: - language_po = language.po(kind) - language_xml = language.xml(kind) - - action = self.w.begin(language_po) - - language_data = None - if not language_xml.exists(): - if require_translation: - # It's easily possible that say a arrays.xml only - # exists in values/, but not in values-xx/. - action.done('skipped') - action.message('%s doesn\'t exist' % language_po.rel, - 'warning') - continue - else: - language_data = read_xml(action, language_xml, language=language) - if not language_data: - # File was invalid - continue - - template_data = read_xml(action, self.env.default.xml(kind)) - if template_data is False: - # File was invalid - continue - - yield action, language_po, template_data, language_data, [language_xml] - - def yield_languages(self, env, source='android'): - if env.options.language: - for code in env.options.language: - if code == '-': - # This allows specifying - to only build the template - continue - language = resolve_locale(code, env) - if language: - yield language - - else: - for l in list_languages(source, env, self.w): - yield l - - def execute(self): - env = self.env - - # First, make sure the templates exist. This makes the "init" - # command everything needed to bootstrap. - # TODO: Test that this happens. - something_done = self.generate_templates(update=False) - - # Only show [exists] actions if a specific language was requested. - show_exists = not bool(env.options.language) - - for language in self.yield_languages(env): - # For each language, generate a .po file. In case a language - # already exists (that is, it's xml files exist, use the - # existing translations for the new gettext catalog). - for (action, - target_po, - template_data, - lang_data, - lang_files) in self._iterate(language, require_translation=False): - if self.generate_po(target_po, template_data, action, - lang_data, lang_files, - update=False, - ignore_exists=show_exists): - something_done = True - - # Also for each language, generate the empty .xml resource files. - # This will make us pick up the language on subsequent runs. - for kind in self.env.xmlfiles: - if write_file(self, language.xml(kind), - """\n\n""", - update=False, ignore_exists=show_exists): - something_done = True - - if not something_done: - self.w.action('info', 'Nothing to do.', 'default') - - -class ExportCommand(InitCommand): - """The export command. - - Inherits from ``InitCommand`` to be able to use ``generate_templates``. - Both commands need to write the templates. - """ - - @classmethod - def setup_arg_parser(cls, parser): - parser.add_argument( - 'language', nargs='*', - help='Language code to export. If not given, all ' + - 'initialized languages will be exported.') - - def execute(self): - env = self.env - w = self.w - - # First, always update the template files. Note that even if - # template generation is disabled, we still need to have the - # catalogs at least in memory for the updating process later on. - # - # TODO: Do we really want to regenerate the templates every - # time, or should the user be able to set fixed meta data, and - # we simply merge subsequent updates in? - self.generate_templates() - - initial_warning = False - - for language in self.yield_languages(env, 'gettext'): - for kind in self.env.xmlfiles: - target_po = language.po(kind) - if not target_po.exists(): - w.action('skipped', target_po) - w.message('File does not exist yet. ' + - 'Use the \'init\' command.') - initial_warning = True - continue - - action = w.begin(target_po) - # If we do not provide a locale, babel will consider this - # catalog a template and always write out the default - # header. It seemingly does not consider the "Language" - # header inside the file at all, and indeed deletes it. - # TODO: It deletes all headers it doesn't know, and - # overrides others. That sucks. - - # Pontoon creates folders like zh-tw, but babel expects zh_tw - locale = language.code.replace('-', '_') - - lang_catalog = read_catalog(target_po, locale=locale) - catalog, _ = self.make_or_get_template(kind, action) - if catalog is None: - # Something went wrong parsing the catalog - continue - lang_catalog.update(catalog, - no_fuzzy_matching=not env.config.enable_fuzzy_matching) - - # Making monkey patching: getting values from obsolete values and - # setting them as the new ones while marking message fuzzy - for message in lang_catalog: - for key in lang_catalog.obsolete: - if key == "": - continue - if message.context == key[1]: - obsolete_message = lang_catalog.obsolete[key] - message.string = obsolete_message.string - message.flags.add('fuzzy') - # Clearing obsolete messages - if env.config.clear_obsolete: - lang_catalog.obsolete.clear() - - # Set the correct plural forms. - current_plurals = lang_catalog.plural_forms - convert.set_catalog_plural_forms(lang_catalog, language) - if lang_catalog.plural_forms != current_plurals: - action.message( - 'The Plural-Forms header of this catalog ' - 'has been updated to what android2po ' - 'requires for plurals support. See the ' - 'README for more information.', 'warning') - - # TODO: Should we include previous? - write_file(self, target_po, - catalog2string(lang_catalog, include_previous=False), - action=action) - - if initial_warning: - print("") - print(colored("Warning: One or more .po files were skipped " + - "because they did not exist yet. Use the 'init' command " + - "to generate them for the first time.", - color='magenta', attrs=['bold'])) - - -class ImportCommand(Command): - """The import command. - """ - - def process(self, language): - """Process importing the given language. - """ - - # In order to implement the --require-min-complete option, we need - # to first determine the translation status across all .po catalogs - # for this language. We can keep the catalogs in memory because we - # will need them later anyway. - catalogs = {} - count_total = 0 - count_translated = 0 - for kind in self.env.xmlfiles: - language_po = language.po(kind) - if not language_po.exists(): - continue - catalogs[kind] = catalog = read_catalog(language_po) - catalog.language = language - ntotal, ntrans, nfuzzy = get_catalog_counts(catalog) - count_total += ntotal - count_translated += ntrans - if self.env.config.ignore_fuzzy: - count_translated -= nfuzzy - - # Compare our count with what is required, if anything. - skip_due_to_incomplete = False - min_required = self.env.config.min_completion - if count_total == 0: - actual_completeness = 1 - else: - actual_completeness = count_translated / float(count_total) - if min_required: - skip_due_to_incomplete = actual_completeness < min_required - - # Now loop through the list of target files, and either create - # them, or print a status message for each indicating that they - # were skipped. - for kind in self.env.xmlfiles: - language_xml = language.xml(kind) - language_po = language.po(kind) - action = self.w.begin(language_xml) - - if skip_due_to_incomplete: - # TODO: Creating a catalog object here is kind of clunky. - # Idially, we'd refactor convert.py so that we can use a - # dict to represent a resource XML file. - xmldata = po2xml(self.env, action, Catalog(locale=language.code)) - write_file(self, language_xml, xml2string(xmldata, action), - action=False) - action.done('skipped', status=('%s catalogs aren\'t ' - 'complete enough - %.2f done' % ( - language.code, - actual_completeness))) - continue - - if not language_po.exists(): - action.done('skipped') - self.w.message('%s doesn\'t exist' % language_po.rel, 'warning') - continue - - content = xml2string(po2xml(self.env, action, catalogs[kind]), action) - write_file(self, language_xml, content, action=action) - - def execute(self): - for language in list_languages('gettext', self.env, self.w): - self.process(language) diff --git a/tools/l10n/android2po/config.py b/tools/l10n/android2po/config.py deleted file mode 100644 index 9a75605f208..00000000000 --- a/tools/l10n/android2po/config.py +++ /dev/null @@ -1,162 +0,0 @@ -from os import path -import argparse - -__all__ = ('Config',) - - -def percentage(string): - errstr = "must be a float between 0 and 1, not %r" % string - try: - value = float(string) - except ValueError: - raise argparse.ArgumentTypeError(errstr) - if value < 0 or value > 1: - raise argparse.ArgumentTypeError(errstr) - return value - - -class Config(object): - """Defines all the options supported by our configuration system. - """ - OPTIONS = ( - { - 'name': 'android', - 'help': 'Android resource directory ($PROJECT/res by default)', - 'dest': 'resource_dir', - 'kwargs': {'metavar': 'DIR'} - # No default, and will not actually be stored on the config object. - }, - { - 'name': 'gettext', - 'help': 'directory containing the .po files ($PROJECT/locale by default)', - 'dest': 'gettext_dir', - 'kwargs': {'metavar': 'DIR'} - # No default, and will not actually be stored on the config object. - }, - { - 'name': 'groups', - 'help': 'process the given default XML files (for example ' + - '"strings arrays"); by default all files which contain ' + - 'string resources will be used', - 'dest': 'groups', - 'default': [], - 'kwargs': {'nargs': '+', 'metavar': 'GROUP'} - }, - { - 'name': 'no-template', - 'help': 'do not generate a .pot template file on export', - 'dest': 'no_template', - 'default': False, - 'kwargs': {'action': 'store_true'} - }, - { - 'name': 'template', - 'help': 'filename to use for the .pot file(s); may contain the ' + - '%%(domain)s and %%(group)s variables', - 'dest': 'template_name', - 'default': '', - 'kwargs': {'metavar': 'NAME'} - }, - { - 'name': 'ignore', - 'help': 'ignore the given message; can be given multiple times; ' + - 'regular expressions can be used if putting the value ' + - 'inside slashes (/match/)', - 'dest': 'ignores', - 'default': [], - 'kwargs': {'metavar': 'MATCH', 'action': 'append', 'nargs': '+'} - }, - { - 'name': 'ignore-fuzzy', - 'help': 'during import, ignore messages marked as fuzzy in .po files', - 'dest': 'ignore_fuzzy', - 'default': False, - 'kwargs': {'action': 'store_true'} - }, - { - 'name': 'require-min-complete', - 'help': 'ignore a language\'s .po file(s) completely if there ' + - 'aren\'t at least the given percentage of translations', - 'dest': 'min_completion', - 'default': 0, - 'kwargs': {'metavar': 'FLOAT', 'type': percentage} - }, - { - 'name': 'domain', - 'help': 'gettext po domain to use, affects the .po filenames', - 'dest': 'domain', - 'default': None, - }, - { - 'name': 'layout', - 'help': 'how and where .po files are stored; may be "default", ' + - '"gnu", or a custom path using the variables %%(locale)s ' + - '%%(domain)s and optionally %%(group)s. E.g., ' + - '"%%(group)s-%%(locale)s.po" will write to "strings-es.po" ' + - 'for Spanish in strings.xml.', - 'dest': 'layout', - 'default': 'default', - }, - { - 'name': 'enable-fuzzy-matching', - 'help': 'enable fuzzy matching during export command. When it is enabled ' + - 'android2po will automatically add translations for new strings. ' + - 'by default this behaviour is turned off', - 'dest': 'enable_fuzzy_matching', - 'default': False, - 'kwargs': {'action': 'store_true'} - }, - { - 'name': 'clear-obsolete', - 'help': 'during export do not add obsolete strings to the generated .po files', - 'dest': 'clear_obsolete', - 'default': True, - 'kwargs': {'action': 'store_true'} - } - ) - - def __init__(self): - """Initialize all configuration values with a default. - - It is important that we do this here manually, rather than relying - on the "default" mechanism of argparse, because we have multiple - potential congiguration sources (command line, config file), and - we don't want defaults to override actual values. - - The attributes we define here are also used to determine - which command line options passed should be assigned to this - object, and which should be exposed via a separate ``options`` - namespace. - """ - for optdef in self.OPTIONS: - if 'default' in optdef: - setattr(self, optdef['dest'], optdef['default']) - - @classmethod - def setup_arguments(cls, parser): - """Setup our configuration values as arguments in the ``argparse`` - object in ``parser``. - """ - for optdef in cls.OPTIONS: - names = ('--%s' % optdef.get('name'),) - kwargs = { - 'help': optdef.get('help', None), - 'dest': optdef.get('dest', None), - # We handle defaults ourselves. This is actually important, - # or defaults from one config source may override valid - # values from another. - 'default': argparse.SUPPRESS, - } - kwargs.update(optdef.get('kwargs', {})) - parser.add_argument(*names, **kwargs) - - @classmethod - def rebase_paths(cls, config, base_path): - """Make those config values that are paths relative to - ``base_path``, because by default, paths are relative to - the current working directory. - """ - for name in ('gettext_dir', 'resource_dir'): - value = getattr(config, name, None) - if value is not None: - setattr(config, name, path.normpath(path.join(base_path, value))) diff --git a/tools/l10n/android2po/convert.py b/tools/l10n/android2po/convert.py deleted file mode 100644 index 12146cc3319..00000000000 --- a/tools/l10n/android2po/convert.py +++ /dev/null @@ -1,948 +0,0 @@ -"""This module does the hard work of converting. - -It uses a simply dict-based memory representation of Android XML string -resource files, via the ``ResourceTree`` class. The .po files are -represented in memory via Babel's ``Catalog`` class. - -The process thus is: - - read_xml() -> ResourceTree -> xml2po() -> Catalog -> po2xml - -> ResourceTree -> write_xml() - -""" - -from __future__ import unicode_literals - -from itertools import chain -from collections import namedtuple -from lxml import etree -from babel.messages import Catalog -from babel.plural import _plural_tags as PLURAL_TAGS -try: - from collections import OrderedDict -except ImportError: # pragma: no cover - from ordereddict import OrderedDict - -__all__ = ('xml2po', 'po2xml', 'read_xml', 'write_xml', - 'set_catalog_plural_forms', 'InvalidResourceError',) - - -class InvalidResourceError(Exception): - pass - - -class UnsupportedResourceError(Exception): - """A resource in a XML file can't be processed. - """ - def __init__(self, reason): - self.reason = reason - - -WHITESPACE = ' \n\t' # Whitespace that we collapse -EOF = None - - -# Some AOSP projects like to include xliff:* tags to annotate -# strings with more information for translators. This is actually harder -# to support than it might look like: We want the translators to see at -# least a tag called "xliff", not the namespace URIs, but we currently -# don't have a way to define namespaces in the .po files (comments?), -# so in order to properly generate an XML on import, we can only deal -# with a fixed list of namespace that we now about. -KNOWN_NAMESPACES = { - 'urn:oasis:names:tc:xliff:document:1.2': 'xliff', -} - - -# The methods here sometimes need to notify the caller about warnings -# processing on; this is why they all take a ``warn_func`` argument. -# By default, if no warnfunc is passed, this dummy will be used. -def dummy_warn(message, severity=None): - return None - - -# These classes are used for the memory representation of an Android -# string resource file. ``ResourceTree`` holds ``StringArray``, -# ``Plurals`` and ``Translation`` objects, and ``StringArray`` and -# ``Plurals`` can also hold ``Translation`` objects. -class ResourceTree(OrderedDict): - language = None - - def __init__(self, language=None): - OrderedDict.__init__(self) - self.language = language - - -class StringArray(list): - pass - - -class Plurals(dict): - pass - - -Translation = namedtuple('Translation', ['text', 'comments', 'formatted']) - - -def get_element_text(tag, name, warnfunc=dummy_warn): - """Return a tuple of the contents of the lxml ``element`` with the - Android specific stuff decoded and whether the text includes - formatting codes. - - "Contents" isn't just the text; it handles nested HTML tags as well. - """ - def convert_text(text): - """This is called for every distinct block of text, as they - are separated by tags. - - It handles most of the Android syntax rules: quoting, escaping, - collapsing duplicate whitespace etc. - """ - # '<' and '>' as literal characters inside a text need to be - # escaped; this is because we need to differentiate them to - # actual tags inside a resource string which we write to the - # .po file as literal '<', '>' characters. As a result, if the - # user puts < inside his Android resource file, this is how - # it will end up in the .po file as well. - # We only do this for '<' and '<' right now, which is of course - # a hack. We'd need to process at least & as well, because - # right now '<' and '&lt;' both generate the same on - # import. However, if we were to do that, a simple non-HTML - # text like "FAQ & Help" would end up us "FAQ & Help" in - # the .po - not particularly nice. - # TODO: I can see two approaches to solve this: Handle things - # differently depending on whether there are nested tags. We'd - # be able to handle both '&lt;' in a HTML string and output - # a nice & character in a plaintext string. - # Option 2: It might be possible to note the type of encoding - # we did in a .po comment. That would even allow us to present - # a string containing tags encoded using entities (but not actual - # nested XML tags) using plain < and > characters in the .po - # file. Instead of a comment, we could change the import code - # to require a look at the original resource xml file to - # determine which kind of encoding was done. - text = text.replace('<', '<') - text = text.replace('>', ">") - - # We need to collapse multiple whitespace while paying - # attention to Android's quoting and escaping. - space_count = 0 - active_quote = False - active_percent = False - active_escape = False - formatted = False - i = 0 - text = list(text) + [EOF] - while i < len(text): - c = text[i] - - # Handle whitespace collapsing - if c is not EOF and c in WHITESPACE: - space_count += 1 - elif space_count > 1: - # Remove duplicate whitespace; Pay attention: We - # don't do this if we are currently inside a quote, - # except for one special case: If we have unbalanced - # quotes, e.g. we reach eof while a quote is still - # open, we *do* collapse that trailing part; this is - # how Android does it, for some reason. - if not active_quote or c is EOF: - # Replace by a single space, will get rid of - # non-significant newlines/tabs etc. - text[i-space_count: i] = ' ' - i -= space_count - 1 - space_count = 0 - elif space_count == 1: - # At this point we have a single whitespace character, - # but it might be a newline or tab. If we write this - # kind of insignificant whitespace into the .po file, - # it will be considered significant on import. So, - # make sure that this kind of whitespace is always a - # standard space. - text[i-1] = ' ' - space_count = 0 - else: - space_count = 0 - - # Handle quotes - if c == '"' and not active_escape: - active_quote = not active_quote - del text[i] - i -= 1 - - # If the string is run through a formatter, it will have - # percentage signs for String.format - if c == '%' and not active_escape: - active_percent = not active_percent - elif not active_escape and active_percent: - formatted = True - active_percent = False - - # Handle escapes - if c == '\\': - if not active_escape: - active_escape = True - else: - # A double-backslash represents a single; - # simply deleting the current char will do. - del text[i] - i -= 1 - active_escape = False - else: - if active_escape: - # Handle the limited amount of escape codes - # that we support. - # TODO: What about \r, or \r\n? - if c is EOF: - # Basically like any other char, but put - # this first so we can use the ``in`` operator - # in the clauses below without issue. - pass - elif c == 'n': - text[i-1: i+1] = '\n' # an actual newline - i -= 1 - elif c == 't': - text[i-1: i+1] = '\t' # an actual tab - i -= 1 - elif c in '"\'@': - text[i-1: i] = '' # remove the backslash - i -= 1 - elif c == 'u': - # Unicode sequence. Android is nice enough to deal - # with those in a way which let's us just capture - # the next 4 characters and raise an error if they - # are not valid (rather than having to use a new - # state to parse the unicode sequence). - # Exception: In case we are at the end of the - # string, we support incomplete sequences by - # prefixing the missing digits with zeros. - # Note: max(len()) is needed in the slice due to - # trailing ``None`` element. - max_slice = min(i+5, len(text)-1) - codepoint_str = "".join(text[i+1: max_slice]) - if len(codepoint_str) < 4: - codepoint_str = "0" * (4-len(codepoint_str)) + codepoint_str - print(repr(codepoint_str)) - try: - # We can't trust int() to raise a ValueError, - # it will ignore leading/trailing whitespace. - if not codepoint_str.isalnum(): - raise ValueError(codepoint_str) - codepoint = chr(int(codepoint_str, 16)) - except ValueError: - raise UnsupportedResourceError('bad unicode escape sequence') - - text[i-1: max_slice] = codepoint - i -= 1 - else: - # All others, remove, like Android does as well. - # However, Android does so silently, we show a - # warning so the dev can fix the problem. - warnfunc(('Resource "%s": removing unsupported ' - 'escape sequence "%s"') % ( - name, "".join(text[i-1: i+1])), 'warning') - text[i-1: i+1] = '' - i -= 1 - active_escape = False - - i += 1 - - # Join the string together again, but w/o EOF marker - return "".join(text[:-1]), formatted - - def get_tag_name(elem): - """For tags without a namespace, returns ("tag", None). - For tags with a known-namespace, returns ("prefix:tag", None). - For tags with an unknown-namespace, returns ("tag", ("prefix", "ns")) - """ - if elem.prefix: - namespace = elem.nsmap[elem.prefix] - raw_name = elem.tag[elem.tag.index('}')+1:] - if namespace in KNOWN_NAMESPACES: - return "%s:%s" % (KNOWN_NAMESPACES[namespace], raw_name), None - return "%s:%s" % (elem.prefix, raw_name), (elem.prefix, namespace) - return elem.tag, None - - # We need to recreate the contents of this tag; this is more - # complicated than you might expect; firstly, there is nothing - # built into lxml (or any other parser I have seen for that - # matter). While it is possible to use ``etree.tostring`` - # to render this tag and it's children, this still would give - # us valid XML code; when in fact we want to decode everything - # XML (including entities), *except* tags. Much more than that - # though, the processing rules the Android xml format needs - # require custom processing anyway. - value = "" - formatted = False - for ev, elem in etree.iterwalk(tag, events=('start', 'end')): - is_root = elem == tag - has_children = len(tag) > 0 - if ev == 'start': - if not is_root: - # Take care of the tag name, namespace and attributes. - # Since we can't store namespace urls in a .po file, dealing - # with (unknown) namespaces requires generating a xmlns - # attribute. - # TODO: We are currently not dealing correctly with - # attribute values that need escaping. - tag_name, to_declare = get_tag_name(elem) - params = ["%s=\"%s\"" % (k, v) for k, v in list(elem.attrib.items())] - if to_declare: - name, url = to_declare - params.append('xmlns:%s="%s"' % (name, url)) - params_str = " %s" % " ".join(params) if params else "" - value += "<%s%s>" % (tag_name, params_str) - if elem.text is not None: - t = elem.text - raw = etree.tostring(elem) - # Leading/Trailing whitespace is removed completely - # ONLY if there are no nested tags. Handle this before - # calling ``convert_text``, so that whitespace - # protecting quotes can still be considered. - if is_root and not has_children and len(tag) == 0: - t = t.strip(WHITESPACE) - - # Resources that start with @ reference other resources. - # While we aren't particularily interested in converting - # those, we also can't do it right now because we wouldn't - # be able to differ between literal @ characters and the - # reference syntax during import. - # - # While it may seem a bit early to deal with this here, we - # have no choice, because the caller needs *some* way of - # differentating between an escaped literal '@' and this - # kind of resource-reference. Since we unescape literals, - # we need to do something with the reference-@. - if is_root and not has_children and t and t[0] == '@': - raise UnsupportedResourceError( - 'resource references (%s) are not supported' % t) - - if "" % tag_name - if elem.tail is not None: - converted_value, elem_formatted = convert_text(elem.tail) - if elem_formatted: - formatted = True - value += converted_value - - # Babel can't handle empty msgids, even when using a unique context; - # not sure if this is a general gettext limitation, but it's not - # unlikely that other tools would have problems, so it's for the better - # in any case. - if value == '': - raise UnsupportedResourceError('empty resources not supported') - return value, formatted - - -def read_xml(xml_file, language=None, warnfunc=dummy_warn): - """Load all resource names from an Android strings.xml resource file. - - The result is a ``ResourceTree`` instance. - """ - result = ResourceTree(language) - comment = [] - - parser = etree.XMLParser(strip_cdata=False) - - try: - doc = etree.parse(xml_file, parser=parser) - except etree.XMLSyntaxError as e: - raise InvalidResourceError(e) - - for tag in doc.getroot(): - # Collect comments so we can add them to the element that they precede. - if tag.tag == etree.Comment: - comment.append(tag.text) - continue - - # Ignore elements we cannot or should not process - if 'name' not in tag.attrib: - comment = [] - continue - if tag.attrib.get('translatable') == 'false': - comment = [] - continue - - name = tag.attrib['name'] - if name in result: - warnfunc('Duplicate resource id found: %s, ignoring.' % name, - 'warning') - comment = [] - continue - - if tag.tag == 'string': - try: - text, formatted = get_element_text(tag, name, warnfunc) - except UnsupportedResourceError as e: - warnfunc('"%s" has been skipped, reason: %s' % ( - name, e.reason), 'info') - else: - translation = Translation(text, comment, formatted) - result[name] = translation - - elif tag.tag == 'string-array': - result[name] = StringArray() - for child in tag.findall('item'): - try: - text, formatted = get_element_text(child, name, warnfunc) - except UnsupportedResourceError as e: - # XXX: We currently can't handle this, because even if - # we write out a .po file with the proper array - # indices, and items like this one missing, during - # import we still need to write out those items that - # we have now skipped, since the Android format is only - # a simple list of items, i.e. we need to specify the - # fully array, and can't override individual items on - # a per-translation basis. - # - # To fix this, we have two options: Either we support - # annotating gettext messages, in which case we could - # indicate whether or not a message like this was a - # reference and should be escaped or not. Or, better, - # the import process would need to use information from - # the default strings.xml file to fill the vacancies. - warnfunc(('Warning: The array "%s" contains items ' + - 'that can\'t be processed (reason: %s) - ' - 'the array will be incomplete') % - (name, e.reason), 'warning') - else: - translation = Translation(text, comment, formatted) - result[name].append(translation) - - elif tag.tag == 'plurals': - result[name] = Plurals() - for child in tag.findall('item'): - try: - quantity = child.attrib['quantity'] - assert quantity in PLURAL_TAGS - except (IndexError, AssertionError): - warnfunc(('"%s" contains a plural with no or ' + - 'an invalid quantity') % name, 'warning') - else: - try: - text, formatted = get_element_text(child, name, warnfunc) - except UnsupportedResourceError as e: - warnfunc(('Warning: The plural "%s" can\'t ' + - 'be processed (reason: %s) - ' - 'the plural will be incomplete') % - (name, e.reason), 'warning') - else: - translation = Translation(text, comment, formatted) - result[name][quantity] = translation - - # We now have processed a tag. We either added those comments to - # the translation we created based on the tag, or the comments - # relate to a tag we do not support. In any case, dismiss them. - comment = [] - - return result - - -def plural_to_gettext(rule): - """This is a copy of the code of ``babel.plural.to_gettext``. - - We need to use a custom version, because the original only returns - a full plural_forms string, which the Babel catalog object does not - allow us to assign to anything. Instead, we need the expr and the - plural count separately. See http://babel.edgewall.org/ticket/291. - """ - from babel.plural import (PluralRule, _fallback_tag, _plural_tags, - _GettextCompiler) - rule = PluralRule.parse(rule) - - used_tags = rule.tags | set([_fallback_tag]) - _compile = _GettextCompiler().compile - _get_index = [tag for tag in _plural_tags if tag in used_tags].index - - expr = ['('] - for tag, ast in rule.abstract: - expr.append('%s ? %d : ' % (_compile(ast), _get_index(tag))) - expr.append('%d)' % _get_index(_fallback_tag)) - return len(used_tags), ''.join(expr) - - -def set_catalog_plural_forms(catalog, language): - """Set the catalog to use the correct plural forms for the - language. - """ - try: - catalog._num_plurals, catalog._plural_expr = plural_to_gettext( - language.locale.plural_form) - except KeyError: - # Babel/CDLR seems to be lacking this data sometimes, for - # example for "uk"; fortunately, ignoring this is narrowly - # acceptable. - pass - - -def xml2po(resources, translations=None, resfilter=None, warnfunc=dummy_warn): - """Return ``resources`` as a Babel .po ``Catalog`` instance. - - If given, ``translations`` will be used for the translated values. - In this case, the returned value is a 2-tuple (catalog, unmatched), - with the latter being a list of Android string resource names that - are in the translated file, but not in the original. - - Both ``resources`` and ``translations`` must be ``ResourceTree`` - objects, as returned by ``read_xml()``. - - From the application perspective, it will call this function with - a ``translations`` object when initializing a new .po file based on - an existing resource file (the 'init' command). For 'export', this - function is called without translations. It will thus generate what - is essentially a POT file (an empty .po file), and this will be - merged into the existing .po catalogs, as per how gettext usually - """ - assert not translations or translations.language - - catalog = Catalog() - if translations is not None: - catalog.locale = translations.language.locale - # We cannot let Babel determine the plural expr for the locale by - # itself. It will use a custom list of plural expressions rather - # than generate them based on CLDR. - # See http://babel.edgewall.org/ticket/290. - set_catalog_plural_forms(catalog, translations.language) - - for name, org_value in resources.items(): - if resfilter and resfilter(name): - continue - - trans_value = None - if translations: - trans_value = translations.pop(name, trans_value) - - if isinstance(org_value, StringArray): - # a string-array, write as "name:index" - if len(org_value) == 0: - warnfunc("Warning: string-array '%s' is empty" % name, 'warning') - continue - - if not isinstance(trans_value, StringArray): - if trans_value: - warnfunc(('""%s" is a string-array in the reference ' - 'file, but not in the translation.') % - name, 'warning') - trans_value = StringArray() - - for index, item in enumerate(org_value): - item_trans = trans_value[index].text if index < len(trans_value) else '' - - # If the string has formatting markers, indicate it in - # the gettext output - flags = [] - if item.formatted: - flags.append('c-format') - - ctx = "%s:%d" % (name, index) - catalog.add(item.text, item_trans, auto_comments=item.comments, - flags=flags, context=ctx) - - elif isinstance(org_value, Plurals): - # a plurals, convert to a gettext plurals - if len(org_value) == 0: - warnfunc("Warning: plurals '%s' is empty" % name, 'warning') - continue - - if not isinstance(trans_value, Plurals): - if trans_value: - warnfunc(('""%s" is a plurals in the reference ' - 'file, but not in the translation.') % - name, 'warning') - trans_value = Plurals() - - # Taking the Translation objects for each quantity in ``org_value``, - # we build a list of strings, which is how plurals are represented - # in Babel. - # - # Since gettext only allows comments/flags on the whole - # thing at once, we merge the comments/flags of all individual - # plural strings into one. - formatted = False - comments = [] - for _, translation in list(org_value.items()): - if translation.formatted: - formatted = True - comments.extend(translation.comments) - - # For the message id, choose any two plural forms, but prefer - # "one" and "other", assuming an English master resource. - temp = org_value.copy() - singular =\ - temp.pop('one') if 'one' in temp else\ - temp.pop('other') if 'other' in temp else\ - temp.pop(list(temp.keys())[0]) - plural =\ - temp.pop('other') if 'other' in temp else\ - temp[list(temp.keys())[0]] if temp else\ - singular - msgid = (singular.text, plural.text) - del temp, singular, plural - - # We pick the quantities supported by the language (the rest - # would be ignored by Android as well). - msgstr = '' - if trans_value: - allowed_keywords = translations.language.plural_keywords - msgstr = ['' for i in range(len(allowed_keywords))] - for quantity, translation in list(trans_value.items()): - try: - index = translations.language.plural_keywords.index(quantity) - except ValueError: - warnfunc( - ('"plurals "%s" uses quantity "%s", which ' - 'is not supported for this language. See ' - 'the README for an explanation. The ' - 'quantity has been ignored') % - (name, quantity), 'warning') - else: - msgstr[index] = translation.text - - flags = [] - if formatted: - flags.append('c-format') - catalog.add(msgid, tuple(msgstr), flags=flags, - auto_comments=comments, context=name) - - else: - # a normal string - - # If the string has formatting markers, indicate it in - # the gettext output - # TODO DRY this. - flags = [] - if org_value.formatted: - flags.append('c-format') - - catalog.add(org_value.text, trans_value.text if trans_value else '', - flags=flags, auto_comments=org_value.comments, context=name) - - if translations is not None: - # At this point, trans_strings only contains those for which - # no original existed. - return catalog, list(translations.keys()) - else: - return catalog - - -def write_to_dom(elem_name, value, ref, namespaces=None, warnfunc=dummy_warn): - """Create a DOM object with the tag name ``elem_name``, containing - the string ``value`` formatted according to Android XML rules. - - The result might be a -tag, or a -tag as found as - children of , for example. - - It might feel awkward at first that the Android-XML formatting - does not happen in a separate method, but is part of the creation - of a tag, but due to us having to do certain formatting based on - child DOM elements that ``value`` may include, the two fit - naturally together (see the POSTPROCESS section of this function). - - If one of our supported namespace prefixes is used within nested tags - inside ``value``, the appropriate data is added to the - ``namespaces`` dict, if given, so the caller may generate the - proper declarations. - """ - - loose_parser = etree.XMLParser(recover=True) - - if value is None: - value = '' - - # PREPROCESS - # The translations may contain arbitrary XHTML, which we need - # to inject into the DOM to properly output. That means parsing - # it first. - # This will now get really messy, since certain XML entities - # we have unescaped for the translators convenience, while the - # tag entities < and > we have not, to differentiate them - # from actual nested tags. Is there any good way to restore this - # properly? - # TODO: In particular, the code below will once we do anything - # bit more complicated with entities, like &amp;lt; - value = value.replace('&', '&') - value = value.replace('&lt;', '<') - value = value.replace('&gt;', '>') - - # PARSE - # - # Namespace handling complicates things a bit. We want the value - # we inject to support nested XML with certain supported namespace - # prefixes, but lxml doesn't seem to allow us to predefine those - # (https://answers.launchpad.net/lxml/+question/111660). - # So we use a wrapping element with xmlns attributes that we ignore - # after parsing. - namespace_text = " ".join(['xmlns:%s="%s"' % (prefix, ns) for ns, prefix in list(KNOWN_NAMESPACES.items())]) - value_to_parse = "<%s>%s" % (namespace_text, elem_name, value, elem_name) - try: - elem = etree.fromstring(value_to_parse) - except etree.XMLSyntaxError as e: - elem = etree.fromstring(value_to_parse, loose_parser) - warnfunc(('%s contains invalid XHTML (%s); Falling back to ' - 'loose parser.') % (ref, e), 'warning') - - # Within the generated DOM, search for use of one of our supported - # namespace prefixes, so we can keep track of which namespaces have - # been used. - if namespaces is not None: - for c in elem.iterdescendants(): - if c.prefix: - nsuri = c.nsmap[c.prefix] - if nsuri in KNOWN_NAMESPACES: - namespaces[KNOWN_NAMESPACES[nsuri]] = nsuri - # Then, proceed with the actual element that we wanted to create. - elem = elem[0] - - def quote(text): - """Return ``text`` surrounded by quotes if necessary. - """ - if text is None: - return - - # If there is trailing or leading whitespace, even if it's - # just a single space character, we need quoting. - needs_quoting = text.strip(WHITESPACE) != text - - # Otherwise, there might be collapsible spaces inside the text. - if not needs_quoting: - space_count = 0 - for c in chain(text, [EOF]): - if c is not EOF and c in WHITESPACE: - space_count += 1 - if space_count >= 2: - needs_quoting = True - break - else: - space_count = 0 - - if needs_quoting: - return '"%s"' % text - return text - - def escape(text): - """Escape all the characters we know need to be escaped - in an Android XML file.""" - if text is None: - return - text = text.replace('\\', '\\\\') - text = text.replace('\n', '\\n') - text = text.replace('\t', '\\t') - text = text.replace('\'', '\\\'') - text = text.replace('"', '\\"') - # Strictly speaking, @ only needs to be escaped when - # it's the first character. But, since our target XML - # files are basically generate-only and unlikely to be - # edited by a user, don't bother with pretty. - text = text.replace('@', '\\@') - return text - - # POSTPROCESS - for child_elem in elem.iter(): - # Strictly speaking, we wouldn't want to touch things - # like the root elements tail, but it doesn't matter here, - # since they are going to be empty string anyway. - child_elem.text = quote(escape(child_elem.text)) - child_elem.tail = quote(escape(child_elem.tail)) - - return elem - - -def key_plural_keywords(x): - """Extracts CLDR plural keywords index starting with 'zero' - and ending with 'other'.""" - return PLURAL_TAGS.index(x) if x in PLURAL_TAGS else -1 - - -def po2xml(catalog, with_untranslated=False, resfilter=None, warnfunc=dummy_warn): - """Convert the gettext catalog in ``catalog`` to a ``ResourceTree`` - instance (our in-memory representation of an Android XML resource) - - This currently relies entirely in the fact that we can use the context - of each message to specify the Android resource name (which we need - to do to handle duplicates, but this is a nice by-product). However - that also means we cannot handle arbitrary catalogs. - - The latter would in theory be possible by using the original, - untranslated XML to match up a messages id to a resource name, but - right now we don't support this (and it's not clear it would be - necessary, even). - - If ``with_untranslated`` is given, then strings in the catalog - that have no translation are written out with the original id, - whenever this is safely possible. This does not include string-arrays, - which for technical reasons always must include all elements, and it - does not include plurals, for which the same is true. - """ - - # Validate that the plurals in the .po catalog match those that - # we expect on the Android side per CLDR definition. However, we - # only want to trouble the user with this if plurals are actually - # used. - plural_validation = {'done': False} - - def validate_plural_config(): - if plural_validation['done']: - return - if catalog.num_plurals != len(catalog.language.plural_keywords): - warnfunc(('Catalog defines %d plurals, we expect %d for ' - 'this language. See the README for an ' - 'explanation. plurals have very likely been ' - 'incorrectly written.') % ( - catalog.num_plurals, len(catalog.language.plural_keywords)), 'error') - pass - plural_validation['done'] = True - - xml_tree = ResourceTree(getattr(catalog, 'language', None)) - for message in catalog: - if not message.id: - # This is the header - continue - - if not message.context: - warnfunc(('Ignoring message "%s": has no context; somebody other ' + - 'than android2po seems to have added to this ' + - 'catalog.') % message.id, 'error') - continue - - if resfilter and resfilter(message): - continue - - # Both string and id will contain a tuple of this is a plural - value = message.string or message.id - - # A colon indicates a string array - if ':' in message.context: - # Collect all the strings of this array with their indices, - # so when we're done processing the whole catalog, we can - # sort by index and restore the proper array order. - name, index = message.context.split(':', 2) - index = int(index) - xml_tree.setdefault(name, StringArray()) - while index >= len(xml_tree[name]): - xml_tree[name].append(None) # fill None for missing indices - if xml_tree[name][index] is not None: - warnfunc(('Duplicate index %s in array "%s"; ignoring ' + - 'the message. The catalog has possibly been ' + - 'corrupted.') % (index, name), 'error') - xml_tree[name][index] = value - - # A plurals message - elif isinstance(message.string, tuple): - validate_plural_config() - - # Untranslated: Do not include those even with with_untranslated - # is enabled - this is because even if we could put the plural - # definition from the master resource here, it wouldn't make - # sense in the context of another language. Instead, let access - # to the untranslated master version continue to work. - if not any(message.string): - continue - - # We need to work with ``message.string`` directly rather than - # ``value``, since ``message.id`` will only be a 2-tuple made - # up of the msgid and msgid_plural definitions. - xml_tree[message.context] = Plurals([ - (k, None) for k in catalog.language.plural_keywords]) - for index, keyword in enumerate(catalog.language.plural_keywords): - # Assume each keyword matches one index. - try: - xml_tree[message.context][keyword] = message.string[index] - except IndexError: - # Plurals are not matching up, validate_plural_config() - # has already raised a warning. - break - - # A standard string. - else: - if not message.string and not with_untranslated: - # Untranslated. - continue - xml_tree[message.context] = value - - return xml_tree - - -# Code from: -# https://stackoverflow.com/questions/4624062/get-all-text-inside-a-tag-in-lxml -# -# Get all child nodes (including text nodes) and join them to a string -def stringify_children(node): - from lxml.etree import tostring - from itertools import chain - parts = ([node.text] + - list(chain(*([tostring(c, with_tail=False), c.tail] for c in node.getchildren()))) + - [node.tail]) - # filter removes possible Nones in texts and tails - return ''.join(filter(None, parts)) - -def write_xml(tree, warnfunc=dummy_warn): - """Takes a ``ResourceTree`` (our in-memory representation of an Android - XML resource) and returns a XML DOM (via an etree.Element). - """ - # Convert the xml tree we've built into an actual Android XML DOM. - root_tags = [] - namespaces_used = {} - for name, value in tree.items(): - if isinstance(value, StringArray): - # string-array - first, sort by index - array_el = etree.Element('string-array') - array_el.attrib['name'] = name - for i, v in enumerate(value): - item_el = write_to_dom( - 'item', v, '"%s" index %d' % (name, i), namespaces_used, - warnfunc) - array_el.append(item_el) - root_tags.append(array_el) - elif isinstance(value, Plurals): - # plurals - plural_el = etree.Element('plurals') - plural_el.attrib['name'] = name - for k in sorted(value, key=key_plural_keywords): - item_el = write_to_dom( - 'item', value[k], '"%s" quantity %s' % (name, k), - namespaces_used, warnfunc) - item_el.attrib["quantity"] = k - plural_el.append(item_el) - root_tags.append(plural_el) - else: - # standard string - string_el = write_to_dom( - 'string', value, '"%s"' % name, namespaces_used, warnfunc) - string_el.attrib['name'] = name - - # If the input text contains html, we need to use CDATA to prevent - # Android's resource processing from turning it into an Android - # formatted spannable string - # Each could contain multiple children. Most of our html resources are in - # a
    , but some have 2

    's in the root, so we convert each child - # to text, concat the children, and then do our html detection. - # We also skip the 0 children case (which signifies 0 html elements in every case), - # since that confuses 'tag in text'. - if len(string_el) > 0: - text = stringify_children(string_el).strip() - - if (tag in text for tag in ['

      ', '

      ', '', '

    • ', '']): - # delete all the existing children, since we're readding them as text - del string_el[:] - string_el.text = etree.CDATA(text) - - root_tags.append(string_el) - - # Generate the root element, define the namespaces that have been - # used across all of our child elements. - root_el = etree.Element('resources', nsmap=namespaces_used) - for e in root_tags: - root_el.append(e) - return root_el diff --git a/tools/l10n/android2po/env.py b/tools/l10n/android2po/env.py deleted file mode 100644 index d00cf282b90..00000000000 --- a/tools/l10n/android2po/env.py +++ /dev/null @@ -1,695 +0,0 @@ -# -*- coding: utf-8 -*- -from __future__ import absolute_import -from __future__ import unicode_literals - -import os -import re -import glob -from argparse import Namespace -from os import path -from babel import Locale -from babel.core import UnknownLocaleError - -from convert import key_plural_keywords -from config import Config -from utils import Path, format_to_re -from convert import read_xml, InvalidResourceError - - -__all__ = ('EnvironmentError', 'IncompleteEnvironment', - 'Environment', 'Language', 'resolve_locale') - - -class EnvironmentError(Exception): - pass - - -class IncompleteEnvironment(EnvironmentError): - pass - - -ANDROID_LOCALE_MAPPING = { - 'from': { - 'in': 'id', - 'iw': 'he', - 'ji': 'yi', - 'zh_CN': 'zh_Hans_CN', - 'zh_HK': 'zh_Hant_HK', - 'zh_TW': 'zh_Hant_TW' - }, - 'to': { - 'id': 'in', - 'he': 'iw', - 'yi': 'ji', - 'zh_Hans_CN': 'zh_CN', - 'zh_Hant_HK': 'zh_HK', - 'zh_Hant_TW': 'zh_TW' - } -} -""" -Android uses locale scheme that differs from one used inside Babel, -so we must provide a mapping between one another. This list is not -full and must be updated to include all such mappings. - -We can not simply ignore middle element in transition from android -to Babel locale mapping. -""" - -MISSING_LOCALES = { - 'ia': { - 'name': "Interlingua", - 'local_name': "Interlingua", - 'plural_rule': 'es', - 'team': 'ia \n' - }, - 'cak': { - 'name': "Kaqchikel", - 'local_name': "Kaqchikel", - 'plural_rule': 'az', - 'team': 'cak \n' - }, - 'zam': { - 'name': "Miahuatlán Zapotec", - 'local_name': "DíɁztè", - 'plural_rule': 'az', - 'team': 'zam \n' - }, - 'trs': { - 'name': "Chicahuaxtla Triqui", - 'local_name': "Triqui", - 'plural_rule': 'az', - 'team': 'trs \n' - }, - 'meh': { - 'name': "Mixteco Yucuhiti", - 'local_name': "Tu´un savi ñuu Yasi'í Yuku Iti", - 'plural_rule': 'id', - 'team': 'meh \n' - }, - 'mix': { - 'name': "Mixtepec Mixtec", - 'local_name': "Tu'un savi", - 'plural_rule': 'id', - 'team': 'mix \n' - }, - 'oc': { - 'name': 'Occitan', - 'local_name': 'occitan', - 'plural_rule': 'fi', - 'team': 'oc \n' - }, - 'an': { - 'name': 'Aragonese', - 'local_name': 'Aragonés', - 'plural_rule': 'fi', - 'team': 'an \n' - }, - 'wo': { - 'name': 'Wolof', - 'local_name': 'Wolof', - 'plural_rule': 'id', - 'team': 'wo \n' - }, - 'tt': { - 'name': 'Tatar', - 'local_name': 'татарча', - 'plural_rule': 'fi', - 'team': 'tt \n' - }, - 'anp': { - 'name': 'Angika', - 'local_name': 'अंगिका', - 'plural_rule': 'bg', - 'team': 'anp \n' - }, - 'tsz': { - 'name': 'Purépecha', - 'local_name': 'p\'urhepecha', - 'plural_rule': 'de', - 'team': 'anp \n' - }, - 'ixl': { - 'name': 'Ixil', - 'local_name': 'ixil', - 'plural_rule': 'de', - 'team': 'anp \n' - }, - 'pai': { - 'name': 'Pai pai', - 'local_name': 'paa ipai', - 'plural_rule': 'lo', - 'team': 'anp \n' - }, - 'quy': { - 'name': 'Quechua Chanka', - 'local_name': 'Chanka Qhichwa', - 'plural_rule': 'pt', - 'team': 'anp \n' - }, - 'ay': { - 'name': 'Aymara', - 'local_name': 'Aimara', - 'plural_rule': 'de', - 'team': 'anp \n' - }, - 'quc': { - 'name': 'K\'iche\'', - 'local_name': 'K\'iche\'', - 'plural_rule': 'de', - 'team': 'anp \n' - }, - 'jv': { - 'name': 'Javanese', - 'local_name': 'Basa Jawa', - 'plural_rule': 'ja', - 'team': 'anp \n' - }, - 'ppl': { - 'name': 'Náhuat Pipil', - 'local_name': 'Náhuat Pipil', - 'plural_rule': 'az', - 'team': 'anp \n' - }, - 'su': { - 'name': 'Sundanese', - 'local_name': 'Basa Sunda', - 'plural_rule': 'ja', - 'team': 'anp \n' - }, - 'hus': { - 'name': 'Huastec', - 'local_name': 'Tének', - 'plural_rule': 'ja', - 'team': 'anp \n' - }, - 'yua': { - 'name': 'Yucatec', - 'local_name': 'Maaya', - 'plural_rule': 'az', - 'team': 'anp \n' - }, - 'ace': { - 'name': 'Acehnese', - 'local_name': 'Basa Acèh', - 'plural_rule': 'id', - 'team': 'anp \n' - }, - 'nv': { - 'name': 'Navajo', - 'local_name': 'Diné Bizaad', - 'plural_rule': 'id', - 'team': 'anp \n' - }, - 'co': { - 'name': 'Corsican', - 'local_name': 'Corsu', - 'plural_rule': 'pt', - 'team': 'anp \n' - }, - 'sn': { - 'name': 'Shona', - 'local_name': 'ChiShona', - 'plural_rule': 'az', - 'team': 'anp \n' - } -} - - -class Language(object): - """Represents a single language.""" - - def __init__(self, code, env=None): - self.code = code - self.env = env - if code and code in MISSING_LOCALES: - self.locale = Locale.parse(MISSING_LOCALES[code]['plural_rule'], sep='-') - elif code: - self.locale = Locale.parse(code, sep='-') - else: - self.locale = None - - def __unicode__(self): # pragma: no cover - return str(self.code) - - def xml(self, kind): - # Android uses a special language code format for the region part - if self.code in ANDROID_LOCALE_MAPPING['to']: - code = ANDROID_LOCALE_MAPPING['to'][self.code] - else: - code = self.code - parts = tuple(code.split('_', 2)) - if len(parts) == 2: - android_code = "%s-r%s" % parts - else: - android_code = "%s" % parts - return self.env.path(self.env.resource_dir, - 'values-%s/%s.xml' % (android_code, kind)) - - def po(self, kind): - filename = self.env.config.layout % { - 'group': kind, - 'domain': self.env.config.domain or 'android', - 'locale': self.code} - return self.env.path(self.env.gettext_dir, filename) - - @property - def plural_keywords(self): - # Sort plural rules properly - ret = list(self.locale.plural_form.rules.keys()) + ['other'] - return sorted(ret, key=key_plural_keywords) - - -class DefaultLanguage(Language): - """A special version of ``Language``, representing the default - language. - - For the Android side, this means the XML files in the values/ - directory. For the gettext side, it means the .pot file(s). - """ - - def __init__(self, env): - super(DefaultLanguage, self).__init__(None, env) - - def __unicode__(self): # pragma: no cover - return '' - - def xml(self, kind): - return self.env.path(self.env.resource_dir, 'values/%s.xml' % kind) - - def po(self, kind): - filename = self.env.config.template_name % { - 'group': kind, - 'domain': self.env.config.domain or 'android', - } - return self.env.path(self.env.gettext_dir, filename) - - -def resolve_locale(code, env): - """Return a ``Language`` instance for a locale code. - - Deals with incorrect Babel locale values.""" - try: - return Language(code, env) - except UnknownLocaleError: - env.w.action('failed', '%s is not a valid locale' % code) - - -def find_project_dir_and_config(): - """Goes upwards through the directory hierarchy and tries to find - either an Android project directory, a config file for ours, or both. - - The latter case (both) can only happen if the config file is in the - root of the Android directory, because once we have either, we stop - searching. - - Note that the two are distinct, in that if a config file is found, - it's directory is not considered a "project directory" from which - default paths can be derived. - - Returns a 2-tuple (project_dir, config_file). - """ - cur = os.getcwd() - - while True: - project_dir = config_file = None - - manifest_path = path.join(cur, 'AndroidManifest.xml') - if path.exists(manifest_path) and path.isfile(manifest_path): - project_dir = cur - - config_path = path.join(cur, '.android2po') - if path.exists(config_path) and path.isfile(config_path): - config_file = config_path - - # Stop once we found either. - if project_dir or config_file: - return project_dir, config_file - - # Stop once we're at the root of the filesystem. - old = cur - cur = path.normpath(path.join(cur, path.pardir)) - if cur == old: - # No further change, we are probably at root level. - # TODO: Is there a better way? Is path.ismount suitable? - # Or we could split the path into pieces by path.sep. - break - - return None, None - - -def find_android_kinds(resource_dir, get_all=False): - """Return a list of Android XML resource types that are in use. - - For this, we simply have a look which xml files exists in the - default values/ resource directory, and return those which - include string resources. - - If ``get_all`` is given, the test for string resources will be - skipped. - """ - kinds = [] - search_dir = path.join(resource_dir, 'values') - for name in os.listdir(search_dir): - filename = path.join(search_dir, name) - if path.isfile(filename) and name.endswith('.xml'): - # We want to support arbitrary xml resource file names, but - # we also need to make sure we only return those which actually - # contain string resources. More specifically, a file named - # my-colors.xml, containing only color resources, should not - # result in a my-colors.po catalog to be created. - # - # We thus attempt to read each file here, see if there are any - # strings in it. If we fail to parse a file, we return it and - # trust that whatever command the user selected will later also - # stumble and show a proper error. - # - # TODO: - # I'm not entirely happy about this. One obvious problem is that - # we are likely to parse these xml files twice, which seems like - # a code smell. One potential solution: Stores the parsed XML - # result directly in memory, with the environment, rather than - # parsing it a second time later. - # - # We could also opt to fail outright if we encounter an invalid - # XML file here, since the error doesn't belong to any "action". - kind = path.splitext(name)[0] - if kind in ('strings', 'arrays') or get_all: - # These kinds are special, they are always supposed to - # contain something translatable, so always include them. - kinds.append(kind) - else: - try: - strings = read_xml(filename) - except InvalidResourceError as e: - raise EnvironmentError('Failed to parse "%s": %s' % (filename, e)) - else: - # If there are any strings in the file, detect as - # a kind of xml file. - if strings: - kinds.append(kind) - return kinds - - -class Environment(object): - """Environment is the main object that holds all the data with - which we run. - - Usage: - - env = Environment() - env.pop_from_config(config) - env.init() - """ - - def __init__(self, writer): - self.w = writer - self.xmlfiles = [] - self.default = DefaultLanguage(self) - self.config = Config() - self.auto_gettext_dir = None - self.auto_resource_dir = None - self.resource_dir = None - self.gettext_dir = None - - # Try to determine if we are inside a project; if so, we a) might - # find a configuration file, and b) can potentially assume some - # default directory names. - self.project_dir, self.config_file = find_project_dir_and_config() - - def _pull_into(self, namespace, target): - """If for a value ``namespace`` there exists a corresponding - attribute on ``target``, then update that attribute with the - values from ``namespace``, and then remove the value from - ``namespace``. - - This is needed because certain options, if passed on the command - line, need nevertheless to be stored in the ``self.config`` - object. We therefore **pull** those values in, and return the - rest of the options. - """ - for name in dir(namespace): - if name.startswith('_'): - continue - if name in target.__dict__: - setattr(target, name, getattr(namespace, name)) - delattr(namespace, name) - return namespace - - def _pull_into_self(self, namespace): - """This is essentially like ``self._pull_info``, but we pull - values into the environment object itself, and in order to avoid - conflicts between option values and attributes on the environment - (for example ``config``), we explicitly specify the values we're - interested in: It's the "big" ones which we would like to make - available on the environment object directly. - """ - for name in ('resource_dir', 'gettext_dir'): - if hasattr(namespace, name): - setattr(self, name, getattr(namespace, name)) - delattr(namespace, name) - return namespace - - def pop_from_options(self, argparse_namespace): - """Apply the set of options given on the command line. - - These means that we need those options that are "configuration" - values to end up in ``self.config``. The normal options will - be made available as ``self.options``. - """ - rest = self._pull_into_self(argparse_namespace) - rest = self._pull_into(rest, self.config) - self.options = rest - - def pop_from_config(self, argparse_namespace): - """Load the values we support into our attributes, remove them - from the ``config`` namespace, and store whatever is left in - ``self.config``. - """ - rest = self._pull_into_self(argparse_namespace) - rest = self._pull_into(rest, self.config) - # At this point, there shouldn't be anything left, because - # nothing should be included in the argparse result that we - # don't consider a configuration option. - ns = Namespace() - assert rest == ns - - def auto_paths(self): - """Try to auto-fill some path values that don't have values yet. - """ - if self.project_dir: - if not self.resource_dir: - self.resource_dir = path.join(self.project_dir, 'res') - self.auto_resource_dir = True - if not self.gettext_dir: - self.gettext_dir = path.join(self.project_dir, 'locale') - self.auto_gettext_dir = True - - def path(self, *pargs): - """Helper that constructs a Path object using the project dir - as the base.""" - return Path(*pargs, base=self.project_dir) - - def init(self): - """Initialize the environment. - - This entails finding the default Android language resource files, - and in the process doing some basic validation. - An ``EnvironmentError`` is thrown if there is something wrong. - """ - # If either of those is not specified, we can't continue. Raise a - # special exception that let's the caller display the proper steps - # on how to proceed. - if not self.resource_dir or not self.gettext_dir: - raise IncompleteEnvironment() - - # It's not enough for directories to be specified; they really - # should exist as well. In particular, the locale/ directory is - # not part of the standard Android tree and thus likely to not - # exist yet, so we create it automatically, but ONLY if it wasn't - # specified explicitely. If the user gave a specific location, - # it seems right to let him deal with it fully. - if not path.exists(self.gettext_dir) and self.auto_gettext_dir: - os.makedirs(self.gettext_dir) - elif not path.exists(self.gettext_dir): - raise EnvironmentError('Gettext directory at "%s" doesn\'t exist.' % - self.gettext_dir) - elif not path.exists(self.resource_dir): - raise EnvironmentError('Android resource direcory at "%s" doesn\'t exist.' % - self.resource_dir) - - # Find the Android XML resources that are our original source - # files, i.e. for example the values/strings.xml file. - groups_found = find_android_kinds(self.resource_dir, - get_all=bool(self.config.groups)) - if self.config.groups: - self.xmlfiles = self.config.groups - _missing = set(self.config.groups) - set(groups_found) - if _missing: - raise EnvironmentError( - 'Unable to find the default XML files for the following groups: %s' % ( - ", ".join(["%s (%s)" % ( - g, path.join(self.resource_dir, 'values', "%s.xml" % g)) for g in _missing]) - )) - else: - self.xmlfiles = groups_found - if not self.xmlfiles: - raise EnvironmentError('no language-neutral string resources found in "values/".') - - # If regular expressions are used as ignore filters, precompile - # those to help speed things along. For simplicity, we also - # convert all static ignores to regexes. - compiled_list = [] - for ignore_list in self.config.ignores: - for ignore in ignore_list: - if ignore.startswith('/') and ignore.endswith('/'): - compiled_list.append(re.compile(ignore[1:-1])) - else: - compiled_list.append(re.compile("^%s$" % re.escape(ignore))) - self.config.ignores = compiled_list - - # Validate the layout option, and resolve magic constants ("gnu") - # to an actual format string. - layout = self.config.layout - multiple_pos = len(self.xmlfiles) > 1 - if not layout or layout == 'default': - if self.config.domain and multiple_pos: - layout = '%(domain)s-%(group)s-%(locale)s.po' - elif self.config.domain: - layout = '%(domain)s-%(locale)s.po' - elif multiple_pos: - layout = '%(group)s-%(locale)s.po' - else: - layout = '%(locale)s.po' - elif layout == 'gnu': - if multiple_pos: - layout = '%(locale)s/LC_MESSAGES/%(group)s-%(domain)s.po' - else: - layout = '%(locale)s/LC_MESSAGES/%(domain)s.po' - else: - # TODO: These tests essentially disallow any advanced - # formatting syntax. While that is unlikely to be used - # or needed, a better way to test for the existance of - # a placeholder would probably be to insert a unique string - # and see if it comes out at the end; or, come up with - # a proper regex to parse. - if '%(locale)s' not in layout: - raise EnvironmentError('--layout lacks %(locale)s variable') - if self.config.domain and '%(domain)s' not in layout: - raise EnvironmentError('--layout needs %(domain)s variable, ', - 'since you have set a --domain') - if multiple_pos and '%(group)s' not in layout: - raise EnvironmentError('--layout needs %%(group)s variable, ' - 'since you have multiple groups: %s' % ( - ", ".join(self.xmlfiles))) - self.config.layout = layout - - # The --template option needs similar processing: - template = self.config.template_name - if not template: - if self.config.domain and multiple_pos: - template = '%(domain)s-%(group)s.pot' - elif self.config.domain: - template = '%(domain)s.pot' - elif multiple_pos: - template = '%(group)s.pot' - else: - template = 'template.pot' - elif '%s' in template and '%(group)s' not in template: - # In an earlier version the --template option only - # supported a %s placeholder for the XML kind. Make - # sure we still support this. - # TODO: Would be nice we if could raise a deprecation - # warning here somehow. That means adding a callback - # to this function. Or, probably we should just make the - # environment aware of the writer object. This would - # simplify other things as well. - template = template.replace('%s', '%(group)s') - else: - # Note that we do not validate %(domain)s here; we expressively - # allow the user to define a template without a domain. - # TODO: See the same case above when handling --layout - if multiple_pos and '%(group)s' not in template: - raise EnvironmentError('--template needs %%(group)s variable, ' - 'since you have multiple groups: %s' % ( - ", ".join(self.xmlfiles))) - self.config.template_name = template - - LANG_DIR = re.compile(r'^values-(\w\w)(?:-r(\w\w))?$') - - def get_android_languages(self): - """Finds the languages that already exist inside the Android - resource directory. - - Return value is a list of ``Language`` instances. - """ - languages = [] - for name in os.listdir(self.resource_dir): - match = self.LANG_DIR.match(name) - if not match: - continue - country, region = match.groups() - pseudo_code = "%s" % country - if region: - pseudo_code += "_%s" % region - if pseudo_code in ANDROID_LOCALE_MAPPING['from']: - code = ANDROID_LOCALE_MAPPING['from'][pseudo_code] - else: - code = pseudo_code - language = resolve_locale(code, self) - if language: - languages.append(language) - return languages - - def get_gettext_languages(self): - """Finds the languages that already exist inside the gettext - directory. - - This is a little more though than on the Android side, since - we give the user a lot of flexibility in configuring how the - .po files are layed out. - - Return value is a list of ``Language`` instances. - """ - - # Build a glob pattern based on the layout. This will enable - # us to easily get a list of files that match the pattern. - glob_pattern = self.config.layout % { - 'domain': self.config.domain, - 'group': '*', - 'locale': '*', - } - - # Temporarily switch to the gettext directory. This allows us - # to simply call glob() using the relative pattern, rather than - # having to deal with making a full path, and then later on - # stripping the full path again for the regex matching, and - # potentially even running into problems when, say, the pattern - # contains references like ../ to a parent directory. - old_dir = os.getcwd() - os.chdir(self.gettext_dir) - try: - list = glob.glob(glob_pattern) - - # We now have a list of matching .po files, but now idea - # which languages they represent, because we don't know - # which part of the filename is the locale. To solve this, - # we build a regular expression from the format string, - # one with a capture group where the locale code should be. - regex = re.compile(format_to_re(self.config.layout)) - - # We then try to match every single file returned by glob. - # In this way, we can build a list of unique locale codes. - languages = {} - for item in list: - m = regex.match(item) - if not m: - continue - code = m.groupdict()['locale'] - if code not in languages: - language = resolve_locale(code, self) - if language: - languages[code] = language - - return languages.values() - finally: - os.chdir(old_dir) diff --git a/tools/l10n/android2po/patch.py b/tools/l10n/android2po/patch.py deleted file mode 100644 index f1c7dce2079..00000000000 --- a/tools/l10n/android2po/patch.py +++ /dev/null @@ -1,123 +0,0 @@ -# -*- coding: utf-8 -*- -from env import MISSING_LOCALES -from cgi import parse_header -from datetime import datetime, time as time_ -from babel import __version__ as VERSION -from babel.core import Locale -from babel._compat import number_types -from babel.dates import format_datetime -from babel.messages import pofile, Catalog -from babel.messages.catalog import _parse_datetime_header -from babel.util import LOCALTZ - - -class PatchedCatalog(Catalog): - def __init__(self, original_locale=None, **kwargs): - super(PatchedCatalog, self).__init__(**kwargs) - self.original_locale = original_locale - - def _get_header_comment(self): - comment = self._header_comment - year = datetime.now(LOCALTZ).strftime('%Y') - if hasattr(self.revision_date, 'strftime'): - year = self.revision_date.strftime('%Y') - comment = comment.replace('PROJECT', self.project) \ - .replace('VERSION', self.version) \ - .replace('YEAR', year) \ - .replace('ORGANIZATION', self.copyright_holder) - if self.original_locale: - comment = comment.replace( - 'Translations template', '%s translations' % MISSING_LOCALES[self.original_locale]['name'] - ) - elif self.locale: - comment = comment.replace('Translations template', '%s translations' % self.locale.english_name) - return comment - - def _set_header_comment(self, string): - self._header_comment = string - - header_comment = property(_get_header_comment, _set_header_comment) - - def _get_mime_headers(self): - headers = [] - headers.append(('Project-Id-Version', - '%s %s' % (self.project, self.version))) - headers.append(('Report-Msgid-Bugs-To', self.msgid_bugs_address)) - headers.append(('POT-Creation-Date', - format_datetime(self.creation_date, 'yyyy-MM-dd HH:mmZ', - locale='en'))) - if isinstance(self.revision_date, (datetime, time_) + number_types): - headers.append(('PO-Revision-Date', - format_datetime(self.revision_date, - 'yyyy-MM-dd HH:mmZ', locale='en'))) - else: - headers.append(('PO-Revision-Date', self.revision_date)) - headers.append(('Last-Translator', self.last_translator)) - if self.locale is not None: - if self.original_locale: - headers.append(('Language', self.original_locale)) - else: - headers.append(('Language', str(self.locale))) - if (self.locale is not None) and ('LANGUAGE' in self.language_team) and not self.original_locale: - headers.append(('Language-Team', self.language_team.replace('LANGUAGE', str(self.locale)))) - elif self.original_locale: - headers.append(('Language-Team', MISSING_LOCALES[self.original_locale]['team'])) - else: - headers.append(('Language-Team', self.language_team)) - if self.locale is not None: - headers.append(('Plural-Forms', self.plural_forms)) - headers.append(('MIME-Version', '1.0')) - headers.append(('Content-Type', - 'text/plain; charset=%s' % self.charset)) - headers.append(('Content-Transfer-Encoding', '8bit')) - headers.append(('Generated-By', 'Babel %s\n' % VERSION)) - return headers - - def _set_mime_headers(self, headers): - for name, value in headers: - name = name.lower() - if name == 'project-id-version': - parts = value.split(' ') - self.project = u' '.join(parts[:-1]) - self.version = parts[-1] - elif name == 'report-msgid-bugs-to': - self.msgid_bugs_address = value - elif name == 'last-translator': - self.last_translator = value - elif name == 'language': - if value in MISSING_LOCALES: - self.locale = Locale.parse(MISSING_LOCALES[value]['plural_rule']) - else: - self.locale = Locale.parse(value) - elif name == 'language-team': - self.language_team = value - elif name == 'content-type': - mimetype, params = parse_header(value) - if 'charset' in params: - self.charset = params['charset'].lower() - elif name == 'plural-forms': - _, params = parse_header(' ;' + value) - self._num_plurals = int(params.get('nplurals', 2)) - self._plural_expr = params.get('plural', '(n != 1)') - elif name == 'pot-creation-date': - self.creation_date = _parse_datetime_header(value) - elif name == 'po-revision-date': - # Keep the value if it's not the default one - if 'YEAR' not in value: - self.revision_date = _parse_datetime_header(value) - - mime_headers = property(_get_mime_headers, _set_mime_headers) - - -def read_po(fileobj, locale=None, domain=None, ignore_obsolete=False, charset=None): - if locale in MISSING_LOCALES: - catalog = PatchedCatalog( - locale=MISSING_LOCALES[locale]['plural_rule'], domain=domain, - charset=charset, original_locale=locale, - copyright_holder="Mozilla", project="Focus for Android" - ) - else: - catalog = PatchedCatalog(locale=locale, domain=domain, charset=charset) - parser = pofile.PoFileParser(catalog, ignore_obsolete) - parser.parse(fileobj) - return catalog diff --git a/tools/l10n/android2po/program.py b/tools/l10n/android2po/program.py deleted file mode 100644 index f049f152cb9..00000000000 --- a/tools/l10n/android2po/program.py +++ /dev/null @@ -1,225 +0,0 @@ -"""Implements the command line interface. -""" - -from __future__ import absolute_import -from __future__ import unicode_literals - -import sys -from os import path -import argparse - -from commands import InitCommand, ExportCommand, ImportCommand, CommandError -from env import IncompleteEnvironment, EnvironmentError, Environment -from config import Config -from utils import Writer - -# Resist the temptation to use "*". It won't work on Python 2.5. -if hasattr(argparse, '__version__') and argparse.__version__ < '1.1': # pragma: no cover - raise RuntimeError('Needs at least argparse 1.1 to function, you are using: %s' % argparse.__version__) - -__all__ = ('main', 'run',) - -COMMANDS = { - 'init': InitCommand, - 'export': ExportCommand, - 'import': ImportCommand -} - - -def parse_args(argv): - """Builds an argument parser based on all commands and configuration - values that we support. - """ - parser = argparse.ArgumentParser( - add_help=True, - description='Convert Android string resources to gettext .po files, an import them back.', - epilog='Written by: Michael Elsdoerfer ' - ) - parser.add_argument('--version', action='version', version="moz1") - - # Create parser for arguments shared by all commands. - base_parser = argparse.ArgumentParser(add_help=False) - group = base_parser.add_mutually_exclusive_group() - group.add_argument('--verbose', '-v', action='store_true', help='be extra verbose') - group.add_argument('--quiet', '-q', action='store_true', help='be extra quiet') - base_parser.add_argument('--config', '-c', metavar='FILE', help='config file to use') - # Add the arguments that set/override the configuration. - group = base_parser.add_argument_group( - 'configuration', - 'Those can also be specified in a configuration file. If given ' - 'here, values from the configuration file will be overwritten.' - ) - Config.setup_arguments(group) - # Add our commands with the base arguments + their own. - subparsers = parser.add_subparsers( - dest="command", title='commands', description='valid commands', help='additional help' - ) - for name, cmdclass in list(COMMANDS.items()): - cmd_parser = subparsers.add_parser(name, parents=[base_parser], add_help=True) - group = cmd_parser.add_argument_group('command arguments') - cmdclass.setup_arg_parser(group) - - return parser.parse_args(argv[1:]) - - -def read_config(in_file): - """Read the config file in ``file``. - - ``file`` may either be a file object, or a filename. - - The config file currently is simply a file with command line options, - each option on a separate line. - - Just for reference purposes, the following ticket should be noted, - which intends to extend argparse with support for configuration files: - http://code.google.com/p/argparse/issues/detail?id=35 - Note however that the current patch doesn't seem to provide an easy - way to make paths in the config relative to the config file location, - as we currently need. - """ - - if hasattr(in_file, 'read'): - lines = in_file.readlines() - if hasattr(in_file, 'name'): - filename = in_file.name - else: - filename = None - else: - # Open the config file and read the arguments. - filename = in_file - f = open(in_file, 'rb') - try: - lines = f.readlines() - finally: - f.close() - - args = filter(lambda x: bool(x), # get rid of '' elements - [i.strip() for i in # get rid of surrounding whitespace - " ".join(filter(lambda x: not x.strip().startswith('#'), - lines) - ).split(" ")]) - - # Use a parser that specifically only supports those options that - # we want to support within a config file (as opposed to all the - # options available through the command line interface). - parser = argparse.ArgumentParser(add_help=False) - Config.setup_arguments(parser) - config, unprocessed = parser.parse_known_args(args) - if unprocessed: - raise CommandError("unsupported config values: %s" % ' '.join(unprocessed)) - - # Post process the config: Paths in the config file should be relative - # to the config location, not the current working directory. - if filename: - Config.rebase_paths(config, path.dirname(filename)) - - return config - - -def make_env_and_writer(argv): - """Given the command line arguments in ``argv``, construct an - environment. - - This entails everything from parsing the command line, parsing - a config file, if there is one, merging the two etc. - - Returns a 2-tuple (``Environment`` instance, ``Writer`` instance). - """ - - # Parse the command line arguments first. This is helpful in - # that any potential syntax errors there will cause us to - # fail before doing anything else. - options = parse_args(argv) - - # Setup the writer verbosity threshold based on the options. - writer = Writer() - if options.verbose: - writer.verbosity = 3 - elif options.quiet: - writer.verbosity = 1 - else: - writer.verbosity = 2 - - env = Environment(writer) - - # Try to load a config file, either if given at the command line, - # or the one that was automatically found. Note that even if a - # config file is used, using the default paths is still supported. - # That is, you can provide some extra configuration values - # through a file, potentially shared across multiple projects, and - # still rely on simply calling the script inside a default - # project's directory hierarchy. - config_file = None - if options.config: - config_file = options.config - env.config_file = config_file - elif env.config_file: - config_file = env.config_file - writer.action('info', "Using auto-detected config file: %s" % config_file) - if config_file: - env.pop_from_config(read_config(config_file)) - - # Now that we have applied the config file, also apply the command - # line options. Those will thus override the config values. - env.pop_from_options(options) - - # Some paths, if we still don't have values for them, can be deducted - # from the project directory. - env.auto_paths() - if env.auto_gettext_dir or env.auto_resource_dir: - # Let the user know we are deducting information from the - # project that we found. - writer.action('info', - "Assuming default directory structure in %s" % env.project_dir) - - # Initialize the environment. This mainly loads the list of - # languages, but also does some basic validation. - try: - env.init() - except IncompleteEnvironment: - if not env.project_dir: - if not env.config_file: - raise CommandError('You need to run this from inside an ' - 'Android project directory, or specify the source and ' - 'target directories manually, either as command line ' - 'options, or through a configuration file') - else: - raise CommandError('Your configuration file does not specify ' - 'the source and target directory, and you are not running ' - 'the script from inside an Android project directory.') - except EnvironmentError as e: - raise CommandError(e) - - # We're done. Just print some info out for the user. - writer.action('info', - "Using as Android resource dir: %s" % env.resource_dir) - writer.action('info', "Using as gettext dir: %s" % env.gettext_dir) - - return env, writer - - -def main(argv): - """The program. - - Returns an error code or None. - """ - try: - # Build an environment from the list of arguments. - env, writer = make_env_and_writer(argv) - try: - cmd = COMMANDS[env.options.command](env, writer) - cmd.execute() - finally: - writer.finish() - if writer.erroneous: - return 1 - return 0 - except CommandError as e: - print('Error:', e) - return 2 - - -def run(): # pragma: no cover - """Simplified interface to main(). - """ - sys.exit(main(sys.argv) or 0) diff --git a/tools/l10n/android2po/requirements.txt b/tools/l10n/android2po/requirements.txt deleted file mode 100644 index f0ef56d116b..00000000000 --- a/tools/l10n/android2po/requirements.txt +++ /dev/null @@ -1,7 +0,0 @@ -# Install using pip install -r requirements.txt - -argparse -babel==2.4 -lxml==4.6.3 -ordereddict -termcolor diff --git a/tools/l10n/android2po/utils.py b/tools/l10n/android2po/utils.py deleted file mode 100644 index cc935cddb4d..00000000000 --- a/tools/l10n/android2po/utils.py +++ /dev/null @@ -1,339 +0,0 @@ -from __future__ import absolute_import -from __future__ import unicode_literals - -from os import getcwd -from sys import stdout -from re import escape as re_escape -from uuid import uuid1 -from locale import getpreferredencoding -from codecs import getwriter -try: - from hashlib import md5 -except ImportError: - import md5 -from os import path -from termcolor import colored - - -__all__ = ('Path', 'Writer', 'file_md5', 'format_to_re',) - - -def format_to_re(format): - """Return the regular expression that matches all possible values - the given Python 2 format string (using %(foo)s placeholders) can - possibly resolve to. - - Each placeholder in the format string is captured in a named group. - - The difficult part here is inserting unescaped regular expression - syntax in place of the format variables, while still properly - escaping the rest. - - See this link for more info on the problem: - http://stackoverflow.com/questions/2654856/python-convert-format-string-to-regular-expression - """ - UNIQ = uuid1().hex - assert UNIQ not in format - - class MarkPlaceholders(dict): - def __getitem__(self, key): - return UNIQ + ('(?P<%s>.*?)' % key) + UNIQ - parts = (format % MarkPlaceholders()).split(UNIQ) - for i in range(0, len(parts), 2): - parts[i] = re_escape(parts[i]) - return ''.join(parts) - - -def file_md5(filename): - """Generate the md5 hash of the given file. - """ - h = md5() - f = open(filename, 'rb') - try: - while True: - # 128 is the md5 digest blocksize - data = f.read(128*10) - if not data: - break - h.update(data) - return h.digest() - finally: - f.close() - - -class Path(str): - """Helper representing a filesystem path that can be "bound" to a base - path. You can then ask it to render as a relative path to that base. - """ - - def __new__(self, *parts, **kwargs): - base = kwargs.pop('base', None) - if kwargs: - raise TypeError() - self.base = base - abs = path.normpath(path.abspath(path.join(*parts))) - return str.__new__(self, abs) - - @property - def rel(self): - """Return this path relative to the base it was bound to. - """ - base = self.base or getcwd() - if not hasattr(path, 'relpath'): # pragma: no cover - # Python < 2.6 doesn't have relpath, and I don't want - # to bother with a wbole bunch of code for this. See - # if we can simply remove the prefix, and if not, 2.5 - # users will have to live with the absolute path. - if self.path.startswith(base): - return self.path[len(base)+1:] - return self.abs - return path.relpath(self, start=base) - - @property - def abs(self): - return self - - def exists(self): - return path.exists(self) - - @property - def dir(self): - return Path(path.dirname(self), base=self.base) - - def hash(self): - return file_md5(self) - - -class Writer(): - """Helps printing messages to the output, in a very particular form. - - Supported are two concepts, "actions" and "messages". A message is - always the child of an action. There is a limited set of action - types (we call them events). Each event and each message may have a - "severity". The severity can determine how a message or event is - rendered (if the terminals supports colors), and will also affect - whether a action or message is rendered at all, depending on verbosity - settings. - - If a message exceeds it's action in severity causing the message to - be visible but the action not, the action will forcably be rendered as - well. For this reason, the class keeps track of the last message that - should have been printed. - - There is also a mechanism which allows to delay printing an action. - That is, you may begin constructing an action and collecting it's - messages, and only later print it out. You would want to do this if - the event type can only be determined after the action is completed, - since it often indicates the outcome. - """ - - # Action types and their default levels - EVENTS = { - 'info': 'info', - 'mkdir': 'default', - 'updated': 'default', - 'unchanged': 'default', - 'skipped': 'warning', - 'created': 'default', - 'exists': 'default', - 'failed': 'error' - } - - # Levels and the minimum verbosity required to show them - LEVELS = {'default': 2, 'warning': 1, 'error': 0, 'info': 3} - - # +2 for [ and ] - # +1 for additional left padding - max_event_len = max([len(k) for k in list(EVENTS.keys())]) + 2 + 1 - - class Action(dict): - def __init__(self, writer, *more, **data): - self.writer = writer - self.messages = [] - self.is_done = False - self.awaiting_promotion = False - dict.__init__(self, {'text': '', 'status': None, 'severity': None}) - self.update(*more, **data) - - def __setitem__(self, name, value): - if name == 'severity': - assert value in Writer.LEVELS, 'Not a valid severity value' - dict.__setitem__(self, name, value) - - def done(self, event, *more, **data): - """Mark this action as done. This will cause it and it's - current messages to be printed, provided they pass the - verbosity threshold, of course. - """ - assert event in Writer.EVENTS, 'Not a valid event type' - self['event'] = event - self.update(*more, **data) - self.writer._print_action(self) - if self in self.writer._pending_actions: - self.writer._pending_actions.remove(self) - self.is_done = True - if self.severity == 'error': - self.writer.erroneous = True - - def update(self, text=None, severity=None, **more_data): - """Update the message with the given data. - """ - if text: - self['text'] = text - if severity: - self['severity'] = severity - dict.update(self, **more_data) - - def message(self, message, severity='info'): - """Print a message belonging to this action. - - If the action is not yet done, this will be added to - an internal queue. - - If the action is done, but was not printed because it didn't - pass the verbosity threshold, it will be printed now. - - By default, all messages use a loglevel of 'info'. - """ - is_allowed = self.writer.allowed(severity) - if severity == 'error': - self.writer.erroneous = True - if not self.is_done: - if is_allowed: - self.messages.append((message, severity)) - elif is_allowed: - if self.awaiting_promotion: - self.writer._print_action(self, force=True) - self.writer._print_message(message, severity) - - @property - def event(self): - return self['event'] - - @property - def severity(self): - sev = self['severity'] - if not sev: - sev = Writer.EVENTS[self.event] - return sev - - def __init__(self, verbosity=LEVELS['default']): - self._current_action = None - self._pending_actions = [] - self.verbosity = verbosity - self.erroneous = False - - # Create a codec writer wrapping stdout - self.stdout = getwriter(self.get_encoding())(stdout) - - @staticmethod - def get_encoding(): - if hasattr(stdout, 'isatty') and stdout.isatty(): - return stdout.encoding - return getpreferredencoding() - - def action(self, event, *a, **kw): - action = Writer.Action(self, *a, **kw) - action.done(event) - return action - - def begin(self, *a, **kw): - """Begin a new action, and return it. The action will not be - printed until you call ``done()`` on it. - - In the meantime, you can attach message to it though, which will - be printed together with the action once it is "done". - """ - action = Writer.Action(self, *a, **kw) - self._pending_actions.append(action) - return action - - def message(self, *a, **kw): - """Attach a message to the last action to be completed. This - includes actions that have not yet been printed (due to not - passing the threshold), but does not include actions that are - not yet marked as 'done'. - """ - self._current_action.message(*a, **kw) - - def finish(self): - """Close down all pending actions that have been began(), but - are not yet done. - - Not the sibling of begin()! - """ - for action in self._pending_actions: - if not action.is_done: - action.done('failed') - self._pending_actions = [] - - def allowed(self, severity): - """Return ``True`` if mesages with this severity pass - the current verbosity threshold. - """ - return self.verbosity >= self.LEVELS[severity] - - def _get_style_for_level(self, severity): - """Return a dict that can be passed as **kwargs to colored(). - """ - # Other colors that work moderately well on both dark and - # light backgrounds and aren't yet used: cyan, green - return { - 'default': {'color': 'blue'}, - 'info': {}, - 'warning': {'color': 'magenta'}, - 'error': {'color': 'red'}, - }.get(severity, {}) - - def get_style_for_action(self, action): - """First looks at the event type to determine a style, then - falls back to severity for good measure. - """ - try: - return { - 'info': {}, # alyways render info in default - 'exists': {'color': 'blue'} - }[action.event] - except KeyError: - return self._get_style_for_level(action.severity) - - def _print_action(self, action, force=False): - """Print the action and all it's attached messages. - """ - if force or self.allowed(action.severity) or action.messages: - self._print_action_header(action) - for m, severity in action.messages: - self._print_message(m, severity) - action.awaiting_promotion = False - else: - # Indicates that this message has not been printed yet, - # and is waiting for a dependent message that needs to - # be printed to trigger it. - action.awaiting_promotion = True - self._current_action = action - - def _print_action_header(self, action): - text = action['text'] - status = action['status'] - if isinstance(text, Path): - # Handle Path instances manually. This doesn't happen - # automatically because we haven't figur out how to make - # that class represent itself through the relative path - # by default, while still returning the full path if it - # is used, say, during an open() operation. - text = text.rel - if status: - text = "%s (%s)" % (text, status) - tag = "[%s]" % action['event'] - - style = self.get_style_for_action(action) - self.stdout.write(colored("%*s" % (self.max_event_len, tag), attrs=['bold'], **style)) - self.stdout.write(" ") - self.stdout.write(colored(text, **style)) - self.stdout.write("\n") - - def _print_message(self, message, severity): - style = self._get_style_for_level(severity) - self.stdout.write(colored(" "*(self.max_event_len+1) + "- %s" % message, - **style)) - self.stdout.write("\n") diff --git a/tools/l10n/check_locales.py b/tools/l10n/check_locales.py deleted file mode 100644 index 97d94d4d0cd..00000000000 --- a/tools/l10n/check_locales.py +++ /dev/null @@ -1,40 +0,0 @@ -#!/usr/bin/python -# -*- coding: utf-8 -*- -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -"""Check imported locales with release locales.""" - -from os import path, listdir -from sys import exit -from locales import SCREENSHOT_LOCALES - -PATH = path.join(path.dirname(path.abspath(__file__)), '../../app/src/main/res/') -IGNORED_DIRECTORIES = ['sw600dp'] - -def check_locales(): - print("Checking for imported locales...") - got_error = False - imported_locales = [ - directory[7:].replace('-r', '-') for directory in listdir(PATH) if directory.startswith('values-') - ] - for ignored in IGNORED_DIRECTORIES: - try: - imported_locales.remove(ignored) - except ValueError: - pass - for locale in imported_locales: - if locale not in SCREENSHOT_LOCALES: - print("Error: * [values-{locale}] missing in SCREENSHOT_LOCALES".format(locale=locale)) - got_error = True - for locale in SCREENSHOT_LOCALES: - if locale not in imported_locales: - print("Warning: * [{locale}] missing in imported locales".format(locale=locale)) - - if got_error: - exit(1) - - -if __name__ == '__main__': - check_locales() diff --git a/tools/l10n/check_translations.py b/tools/l10n/check_translations.py deleted file mode 100644 index ab7600bcc16..00000000000 --- a/tools/l10n/check_translations.py +++ /dev/null @@ -1,86 +0,0 @@ -#!/usr/bin/python -# -*- coding: utf-8 -*- -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -"""Check translation files for missing or wrong number of placeholders.""" - -from os import path, walk -from sys import exit -import xml.etree.ElementTree as ET - - -def etree_to_dict(tree): - d = {} - for element in tree.getchildren(): - if 'name' in element.attrib: - d[element.attrib['name']] = element.text - else: - d.update(etree_to_dict(element)) - return d - - -def missing_target_exception(lang, code): - # there is placeholder in source, but not in target. - print("Error: * [{lang}/strings.xml] missing placeholder in translation, key: {code}".format( - lang=lang, - code=code - )) - -def missing_source_exception(lang, code): - # there is placeholder in target, but not in source. - print("Error: * [{lang}/strings.xml] placeholder missing in source, key: {code}".format( - lang=lang, - code=code - )) - -def count_mismatch_warning(lang, code): - # number of same placeholders used in source and target do not match - print("Warning: * [{lang}/strings.xml] number of placeholders not matching, key: {code}".format( - lang=lang, - code=code - )) - -PATH = path.join(path.dirname(path.abspath(__file__)), '../../app/src/main/res/') -files = [] - -# Make list of all locale xml files -for directory, directories, file_names in walk(PATH): - for file_name in file_names: - if file_name == "strings.xml" and 'values-' in directory: - files.append( - path.join(directory, file_name) - ) -# Read source file -source_xml = ET.parse(path.join(PATH, 'values/strings.xml')) -source = etree_to_dict(source_xml.getroot()) - -got_error = False - -print("Checking for placeholders in translation files...") - -for source_xml in files: - target = etree_to_dict(ET.parse(source_xml).getroot()) - for key in source: - # pass missing translations and empty strings - if key not in target: - continue - if not target[key]: - continue - if not source[key]: - continue - # Check for placeholders - language = source_xml.split('/')[-2] - for placeholder in ['%1$s', '%2$s', '%3$s', '%4$s', '%5$s']: - if placeholder in source[key] and placeholder not in target[key]: - missing_target_exception(language, key) - got_error = True - elif placeholder in target[key] and placeholder not in source[key]: - missing_source_exception(language, key) - got_error = True - elif source[key].count(placeholder) != target[key].count(placeholder): - count_mismatch_warning(language, key) - -if got_error: - exit(1) diff --git a/tools/l10n/create_commits.sh b/tools/l10n/create_commits.sh deleted file mode 100755 index 521debec4fe..00000000000 --- a/tools/l10n/create_commits.sh +++ /dev/null @@ -1,25 +0,0 @@ -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -# Create a separate commit for every locale. - -parent_path=$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P ) -cd "$parent_path/../../l10n-repo" - -git add locales/templates/app.pot -git commit -m "template update: app.pot" - -cd locales - -locale_list=$(find . -mindepth 1 -maxdepth 1 -type d \( ! -iname ".*" \) | sed 's|^\./||g' | sort) -for locale in ${locale_list}; -do - # Exclude templates - if [ "${locale}" != "templates" ] - then - git add ${locale}/app.po - git commit -m "${locale}: Update app.po" - fi -done - diff --git a/tools/l10n/export-strings.sh b/tools/l10n/export-strings.sh deleted file mode 100755 index 8c0f1a38e34..00000000000 --- a/tools/l10n/export-strings.sh +++ /dev/null @@ -1,41 +0,0 @@ -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -# If a command fails then do not proceed and fail this script too. -set -e - -# Go to project root -cd "$(dirname "$0")" -cd ../.. - -# Checkout l10n repository or update already existing checkout -if [ ! -d "l10n-repo" ]; then - git clone https://github.com/mozilla-l10n/focus-android-l10n.git l10n-repo -else - cd l10n-repo - git fetch origin - cd .. -fi - -# Reset the repo to the master state -cd l10n-repo -git reset --hard origin/master -git checkout master -git reset --hard origin/master - -# Create a branch for the export -BRANCH="export-"`date +%Y-%m-%d` -git branch -D $BRANCH || echo "" -git checkout -b $BRANCH -cd .. - -# Export strings and convert them from Android strings.xml files to po files -python tools/l10n/android2po/a2po.py export || echo "Could not export all locales" - -# Create a separate commit for every locale. -tools/l10n/create_commits.sh - -echo "" -echo "Please push and review branch $BRANCH in l10n-repo/" - diff --git a/tools/l10n/fix_locale_folders.sh b/tools/l10n/fix_locale_folders.sh deleted file mode 100755 index e7afcd32d79..00000000000 --- a/tools/l10n/fix_locale_folders.sh +++ /dev/null @@ -1,39 +0,0 @@ -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -# a2po has problems with the folder format of Pontoon and creates resource folders -# like values-es-MX. Android expects values-es-rMX. This script tries to find those -# folders and fixes them. - -parent_path=$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P ) - -cd "$parent_path/../../app/src/main/res/" - - -folder_list=$(find . -maxdepth 1 -type d -iname "values-*-*") -for folder in ${folder_list}; -do - country=$(echo ${folder} | cut -d'-' -f3) - len=${#country} - - if [ "$len" -eq "2" ]; then - prefix=$(echo ${folder} | cut -d'-' -f1,2) - - fixed_folder="${prefix}-r${country}" - - echo "Fixing ${folder} -> ${fixed_folder}" - - # The target folder might already have some data (e.g. our urls.xml), - # hence we only copy newly generated files over (and keep existing - # non-generated files). - if [ ! -d "${fixed_folder}" ]; then - mkdir "${fixed_folder}" - fi - - cp -r "$folder"/* "${fixed_folder}" - - rm -rf "$folder" - fi -done - diff --git a/tools/l10n/import-strings.sh b/tools/l10n/import-strings.sh deleted file mode 100755 index f02df9f06e2..00000000000 --- a/tools/l10n/import-strings.sh +++ /dev/null @@ -1,29 +0,0 @@ -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -# If a command fails then do not proceed and fail this script too. -set -e - -# Go to project root -cd "$(dirname "$0")" -cd ../.. - -# Checkout l10n repository or update already existing checkout -if [ ! -d "l10n-repo" ]; then - git clone https://github.com/mozilla-l10n/focus-android-l10n.git l10n-repo -else - cd l10n-repo - git fetch origin - git reset --hard origin/master - cd .. -fi - -# Import and convert po files in L10N repository to Android strings.xml files -python tools/l10n/android2po/a2po.py import || echo "Could not import all locales" - -# a2po creates wrong folder names for locales with country (e.g. values-es-MX). Rename -# those so that Android can find them (e.g. values-es-rMX) -tools/l10n/fix_locale_folders.sh - -python tools/l10n/check_translations.py diff --git a/tools/taskcluster/android-wait-for-emulator.sh b/tools/taskcluster/android-wait-for-emulator.sh deleted file mode 100755 index 77fddf50c62..00000000000 --- a/tools/taskcluster/android-wait-for-emulator.sh +++ /dev/null @@ -1,22 +0,0 @@ -#!/bin/bash - -# From: https://github.com/mindrunner/docker-android-sdk/blob/master/tools/android-wait-for-emulator.sh -# Originally written by Ralf Kistner , but placed in the public domain - -set +e - -bootanim="" -failcounter=0 -until [[ "$bootanim" =~ "stopped" ]]; do - bootanim=`adb -e shell getprop init.svc.bootanim 2>&1` - echo "$bootanim" - if [[ "$bootanim" =~ "not found" ]]; then - let "failcounter += 1" - if [[ $failcounter -gt 15 ]]; then - echo "Failed to start emulator" - exit 1 - fi - fi - sleep 1 -done -echo "Done" diff --git a/tools/taskcluster/create-pull-request.py b/tools/taskcluster/create-pull-request.py deleted file mode 100644 index 0218a0e770d..00000000000 --- a/tools/taskcluster/create-pull-request.py +++ /dev/null @@ -1,71 +0,0 @@ - # This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -""" -This scripts pushes the passed in branch (argument) to the bot's -repository and creates a pull request for it. -""" - -import os -import re -import subprocess -import sys -import taskcluster - -from github import Github - -# Make sure we have all needed arguments -if len(sys.argv) != 2: - print "Usage", sys.argv[0], "BRANCH" - exit(1) - -# Get token for GitHub bot account from secrets service -secrets = taskcluster.Secrets({'baseUrl': 'http://taskcluster/secrets/v1'}) -data = secrets.get('project/focus/github') -token = data['secret']['botAccountToken'] - -BRANCH = sys.argv[1] -OWNER = 'mozilla-mobile' -USER = 'MickeyMoz' -REPO = 'focus-android' -BASE = 'master' -HEAD = "MickeyMoz:%s" % BRANCH -URL = "https://%s:%s@github.com/%s/%s/" % (USER, token, USER, REPO) - -github = Github(login_or_token=token) -repo = github.get_user(OWNER).get_repo(REPO) - -# Check if there's already an umerged pull request. -for request in repo.get_pulls(state='open'): - if request.user.login == USER: - print "There's already an unmerged pull request. Doing nothing." - exit(0) - -# Push local state to branch -print subprocess.check_output(['git', 'push', URL, BRANCH]) - -# Read the log file and create the pull request body from it -log_path = os.path.join(os.path.dirname(__file__), '../../import-log.txt') -with open(log_path) as log_file: - # Remove color codes from android2po output - ansi_escape = re.compile(r'\x1b[^m]*m') - log = ansi_escape.sub('', log_file.read()) - -body = """ -Automated import %s - -Log: -``` -%s -``` -""" % (BRANCH, log) - -# Create pull request -pull_request = repo.create_pull(title='String import ' + BRANCH, body=body, base=BASE, head=HEAD) -print pull_request - -# Add the [needs merge] label to the pull request -issue = repo.get_issue(pull_request.number) -issue.add_to_labels("needs merge") -print issue diff --git a/tools/taskcluster/import_strings_and_create_pull_request.sh b/tools/taskcluster/import_strings_and_create_pull_request.sh deleted file mode 100755 index c706f8b4dec..00000000000 --- a/tools/taskcluster/import_strings_and_create_pull_request.sh +++ /dev/null @@ -1,30 +0,0 @@ -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -# This script imports the latest strings, creates a commit, pushes -# it to the bot's repository and creates a pull request. - -# If a command fails then do not proceed and fail this script too. -set -e - -# Go to project root -cd "$(dirname "$0")" -cd ../.. - -# Import strings from L10N repository -tools/l10n/import-strings.sh | tee import-log.txt ; test ${PIPESTATUS[0]} -eq 0 - -# Timestamp used in branch name and commit -TIMESTAMP=`date "+%Y%m%d-%H%M%S"` - -# Create a branch and commit local changes -git checkout -b $TIMESTAMP -git add app/src/main/res/ -git commit -m \ - "Import translations from L10N repository ($TIMESTAMP)" \ - --author="MickeyMoz " \ - || { echo "No new translations available"; exit 0; } - -# Create a pull request for the current state of the repo -python tools/taskcluster/create-pull-request.py $TIMESTAMP diff --git a/tools/taskcluster/lib/tasks.py b/tools/taskcluster/lib/tasks.py index 21cd4202cc7..3cc4f7cbd6a 100644 --- a/tools/taskcluster/lib/tasks.py +++ b/tools/taskcluster/lib/tasks.py @@ -140,7 +140,7 @@ def build_push_task(self, signing_task_id, name, description, apks=[], scopes=[] "name": name, "description": description, "owner": "skaspari@mozilla.com", - "source": "https://github.com/mozilla-mobile/focus-android/tree/master/tools/taskcluster" + "source": "https://github.com/mozilla-mobile/focus-android/tree/main/tools/taskcluster" } } diff --git a/tools/taskcluster/release.py b/tools/taskcluster/release.py index ec4c30b3804..301c04abdc5 100644 --- a/tools/taskcluster/release.py +++ b/tools/taskcluster/release.py @@ -38,7 +38,7 @@ def generate_build_task(apks, tag): } artifacts["public/%s" % os.path.basename(apk)] = artifact - checkout = "git fetch origin && git reset --hard origin/master" if tag is None else "git fetch origin && git checkout %s" % (tag) + checkout = "git fetch origin && git reset --hard origin/main" if tag is None else "git fetch origin && git checkout %s" % (tag) assemble_task = 'assembleNightly' diff --git a/tools/taskcluster/schedule-master-build.py b/tools/taskcluster/schedule-main-build.py similarity index 98% rename from tools/taskcluster/schedule-master-build.py rename to tools/taskcluster/schedule-main-build.py index 7b078fbca6e..b398dba9db7 100644 --- a/tools/taskcluster/schedule-master-build.py +++ b/tools/taskcluster/schedule-main-build.py @@ -4,7 +4,7 @@ """ This script will be executed whenever a change is pushed to the -master branch. It will schedule multiple child tasks that build +main branch. It will schedule multiple child tasks that build the app, run tests and execute code quality tools: """ @@ -20,7 +20,7 @@ BRANCH = os.environ.get('MOBILE_HEAD_BRANCH') COMMIT = os.environ.get('MOBILE_HEAD_REV') OWNER = "skaspari@mozilla.com" -SOURCE = "https://github.com/mozilla-mobile/focus-android/tree/master/tools/taskcluster" +SOURCE = "https://github.com/mozilla-mobile/focus-android/tree/main/tools/taskcluster" def generate_build_task(): diff --git a/tools/taskcluster/schedule-screenshots.py b/tools/taskcluster/schedule-screenshots.py deleted file mode 100644 index ed5e52a3b8c..00000000000 --- a/tools/taskcluster/schedule-screenshots.py +++ /dev/null @@ -1,101 +0,0 @@ -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -# This script is executed on taskcluster as a "decision task". It will schedule -# individual tasks for taking screenshots for all locales. - -import datetime -import json -import os -import sys -import taskcluster - -# Add the l10n folder to the system path so that we can import scripts from it. -sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'l10n')) -from locales import SCREENSHOT_LOCALES - -# Get task id and other metadata from the task we are running in (This only works on taskcluster) -TASK_ID = os.environ.get('TASK_ID') - -# The number of locales that we will generate screenshots for per taskcluster task. -LOCALES_PER_TASK = 5 - -def generate_screenshot_task(locales): - """Generate a new task that takes screenshots for the given set of locales""" - parameters = " ".join(locales) - - created = datetime.datetime.now() - expires = taskcluster.fromNow('1 week') - deadline = taskcluster.fromNow('1 day') - - return { - "workerType": "b-linux", - "taskGroupId": TASK_ID, - "expires": taskcluster.stringDate(expires), - "retries": 5, - "created": taskcluster.stringDate(created), - "tags": {}, - "priority": "lowest", - "schedulerId": "taskcluster-github", - "deadline": taskcluster.stringDate(deadline), - "dependencies": [ TASK_ID ], - "routes": [], - "scopes": [], - "requires": "all-completed", - "payload": { - "features": { - "taskclusterProxy": True - }, - "maxRunTime": 7200, - "image": "mozillamobile/focus-android:1.2", - "command": [ - "/bin/bash", - "--login", - "-c", - """ git fetch origin - && git config advice.detachedHead false - && git checkout origin/master - && /opt/focus-android/tools/taskcluster/take-screenshots.sh %s""" % parameters - ], - "artifacts": { - "public": { - "type": "directory", - "path": "/opt/focus-android/fastlane/metadata/android", - "expires": taskcluster.stringDate(expires) - } - }, - "deadline": taskcluster.stringDate(deadline) - }, - "provisionerId": "mobile-1", - "metadata": { - "name": "Screenshots for locales: %s" % parameters, - "description": "Generating screenshots of Focus for Android in the specified locales (%s)" % parameters, - "owner": "skaspari@mozilla.com", - "source": "https://github.com/mozilla-mobile/focus-android/tree/master/tools/taskcluster" - } - } - - -def chunks(locales, n): - """Yield successive n-sized chunks from the list of locales""" - for i in range(0, len(locales), n): - yield locales[i:i + n] - - -if __name__ == "__main__": - print "Task:", TASK_ID - - queue = taskcluster.Queue({ - 'baseUrl': 'http://taskcluster/queue/v1' - }) - - for chunk in chunks(SCREENSHOT_LOCALES, LOCALES_PER_TASK): - taskId = taskcluster.slugId() - print "Create screenshots task (%s) for locales:" % taskId, " ".join(chunk) - - task = generate_screenshot_task(chunk) - print json.dumps(task, indent=4, separators=(',', ': ')) - - result = queue.createTask(taskId, task) - print json.dumps(result) diff --git a/tools/taskcluster/screencap-server.py b/tools/taskcluster/screencap-server.py deleted file mode 100644 index b2f128cb6d6..00000000000 --- a/tools/taskcluster/screencap-server.py +++ /dev/null @@ -1,39 +0,0 @@ -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -# This is a little HTTP server that runs "screencap" on the connected -# Android device via "adb shell". This allows the Android device -# itself to trigger "screencap" via HTTP. -# -# We are running emulators without "-gpu off" and with that we are not -# able to take screenshots via UIAutomator. Other ways of taking -# screenshots from the Android device itself do not capture the full -# screen as the user sees it. - -from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer - -from os import curdir, sep -import subprocess - -class ScreenshotHandler(BaseHTTPRequestHandler): - def do_GET(self): - self.send_response(200) - self.send_header('Content-type', 'text/plain') - self.end_headers() - - print " * Taking screenshot" - - command = "adb shell screencap -p /data/data/org.mozilla.focus.debug/files/temp_screen.png" - process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE) - process.wait() - - self.wfile.write("screenshot, exit=" + str(process.returncode)) - - print " * Done ( exit = ", process.returncode, ")" - -if __name__ == "__main__": - httpd = HTTPServer(('0.0.0.0', 9771), ScreenshotHandler) - print 'Starting httpd...' - httpd.serve_forever() - diff --git a/tools/taskcluster/take-screenshots.sh b/tools/taskcluster/take-screenshots.sh deleted file mode 100755 index 6f6b75e3a03..00000000000 --- a/tools/taskcluster/take-screenshots.sh +++ /dev/null @@ -1,49 +0,0 @@ -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -# This script takes screenshots of Focus in different locales on -# taskcluster. The locales need to be passed to this script, e.g.: -# ./take-screenshots.sh en de-DE fr - -# If a command fails then do not proceed and fail this script too. -set -e - -# Make sure we passed locales to this script. -if (( $# < 1 )); then - echo "No locales passed to this script, e.g.: ./take-screenshots.sh en de-DE fr" - exit 1 -fi - -echo "Taking screenshots for locales: $@" -directory="$(dirname "$0")" - -# Required for fastlane (otherwise it just crashes randomly) -export LC_ALL="en_US.UTF-8" - -# Start emulator (in background) -emulator64-arm -avd test -noaudio -no-window -no-accel -gpu off -verbose & - -# Build and install app & test APKs (while the emulator is booting..) -./gradlew --no-daemon assembleFocusDebug assembleFocusDebugAndroidTest - -# Start our server for running screencap on the emulator host (via HTTP) -python $directory/screencap-server.py & - -# Generate Screengrab configuration -python $directory/generate_screengrab_config.py $@ - -# Wait for emulator to finish booting -/opt/focus-android/tools/taskcluster/android-wait-for-emulator.sh - -# Install app and make sure directory for taking screenshot exists. -adb install -r app/build/outputs/apk/focus/nightly/app-focus-armeabi-v7a-debug.apk -adb shell mkdir /data/data/org.mozilla.focus.debug/files - -# Take screenshots -fastlane screengrab run - -# Resize and optimize all screenshots -cd fastlane -find . -type f -iname "*.png" -print0 | xargs -I {} -0 mogrify -resize 75% "{}" -find . -type f -iname "*.png" -print0 | xargs -I {} -0 optipng -o5 "{}" \ No newline at end of file