New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Filtering Log v1 #11

Closed
ameshkov opened this Issue Aug 24, 2015 · 20 comments

Comments

Projects
None yet
4 participants
@ameshkov
Member

ameshkov commented Aug 24, 2015

View 1 - Filtering Log is stopped

Placeholder
Start button
Warning about performance

Filtering Log Notification

When filtering is started we should show an ongoing notification about this.

View 2 - Filtering Log is started

List of log records.

Show for every log record:
URL
App name
Source domain
Third-Party icon

Log record color indicates if request was blocked or whitelisted.
Red for blocked, green for whitelisted, white bg - for processed.

View 3 - Log Record Details

Tapping on log record opens log record details view.

URL
Type
Source domain
Filter Rule and Filter Subscription name for blocked and whitelisted requests

Buttons:
Open in browser
Block for common requests
Unblock for blocked requests (Unblock creates whitelist rule)
Nothing for requests whitelisted by some filter subsription

View 4 - Filter Rule Creation

Example of such dialog in browser add-on:
http://screencast.com/t/kbttytH1SWj

New rules should be added to user filter.

@ameshkov ameshkov added this to the 2.2 milestone Aug 24, 2015

@Nick-White

This comment has been minimized.

Show comment
Hide comment
@Nick-White

Nick-White Oct 24, 2015

+1 This is the feature I'm waiting the most because there are many ads that go through all the filter subscriptions...

Nick-White commented Oct 24, 2015

+1 This is the feature I'm waiting the most because there are many ads that go through all the filter subscriptions...

@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Oct 26, 2015

Member

For now you can report these to us:
https://github.com/AdguardTeam/ExperimentalFilter/issues

Or you can read AG logs and find out what is not blocked:
https://github.com/AdguardTeam/ExperimentalFilter/wiki/Android

Member

ameshkov commented Oct 26, 2015

For now you can report these to us:
https://github.com/AdguardTeam/ExperimentalFilter/issues

Or you can read AG logs and find out what is not blocked:
https://github.com/AdguardTeam/ExperimentalFilter/wiki/Android

@ameshkov ameshkov modified the milestones: 2.9, 3.0 Feb 20, 2017

@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Feb 27, 2017

Member

@Revertron here is a list of comments for the first filtering log implementation

Filtering log icon

Use this icon for the navigation menu item:
filtering_log_icons.zip

Please let me know what sizes do you need.

Action bar

We need an option to clear the filtering log
Put it in the action bar.
Use the same icon we have in the user filter/whitelist.

"Enabled" should the same size/weigth as action bar title has.

Empty filtering log

Use the same icons and layout as in the case of user filter/whitelist.
Text: Filtering log is empty

Log item

Time and URL color should be of our regular gray color.

Member

ameshkov commented Feb 27, 2017

@Revertron here is a list of comments for the first filtering log implementation

Filtering log icon

Use this icon for the navigation menu item:
filtering_log_icons.zip

Please let me know what sizes do you need.

Action bar

We need an option to clear the filtering log
Put it in the action bar.
Use the same icon we have in the user filter/whitelist.

"Enabled" should the same size/weigth as action bar title has.

Empty filtering log

Use the same icons and layout as in the case of user filter/whitelist.
Text: Filtering log is empty

Log item

Time and URL color should be of our regular gray color.

@Revertron

This comment has been minimized.

Show comment
Hide comment
@Revertron

Revertron Feb 27, 2017

Member

For small icons (as 24dp in UI) we use these sizes: 96, 72, 48, 24, 36 pixels.

Member

Revertron commented Feb 27, 2017

For small icons (as 24dp in UI) we use these sizes: 96, 72, 48, 24, 36 pixels.

@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Feb 27, 2017

Member

@Revertron got it, what color do we need for the navigation menu?

Member

ameshkov commented Feb 27, 2017

@Revertron got it, what color do we need for the navigation menu?

@Revertron

This comment has been minimized.

Show comment
Hide comment
@Revertron

Revertron Feb 27, 2017

Member

Other icons have $757575 color.

Member

Revertron commented Feb 27, 2017

Other icons have $757575 color.

@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Feb 27, 2017

Member

Color indication

Change the line background color to indicate request status.

Regular requests: no indication.
Blocked requests: #fff4f2 (pressed state: #ffb6a9)
Whitelisted requests: #defae5 (pressed state: #98ffb2)

Member

ameshkov commented Feb 27, 2017

Color indication

Change the line background color to indicate request status.

Regular requests: no indication.
Blocked requests: #fff4f2 (pressed state: #ffb6a9)
Whitelisted requests: #defae5 (pressed state: #98ffb2)

@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Feb 27, 2017

Member

Log item details fields should depend on the event type. However, there are some common requirements.

Common requirements

  1. Tap on a field should automatically copy it's value to the clipboard. We should also show a toast about it.
  2. The Status field possible values: Processed, Blocked, Threat, Whitelisted, Redirect. The latter makes sense for Web request type only.
  3. Event type field possible values are Web request, DNS request, HTTPS tunnel
  4. Application field name depends on the app type and can be App, Browser, System app
  5. Filtering rule field is visible in two cases only: if the status is either Blocked or Whitelisted.
  6. Possible buttons in the item details footer: BLOCK, UNBLOCK (raised buttons), OPEN (flat button). OPEN button opens request URL in the default browser. OPEN is available for regular web requests with GET method only.
  7. Remote Address field format is 127.0.0.1:443 for IPv4 and [2001:4860:4860::8888]:443 for IPv6. Use IpAddressUtils.toString() method to get the proper format for it. Do not show this field if value is empty.

Regular web request fields

  • Event type
  • Application
  • Domain
  • Remote Address
  • Status (You should use Redirect status whenever redirectUrl field is not empty.)
  • Filtering rule
  • Request URL
  • Source URL (do not show if empty or null)
  • Redirect URL (show it if status is Redirect)
  • Content type (no need to translate, just use the enum value)
  • Start time (format: 18:45.123) (we don't really need date here)
  • Elapsed time (format: 123 ms)
  • Size (format: ↓123KB ↑5KB)

Please note, that Event type in the case of regular web request also contains HTTP method value.
It should look like:

Value: Web request (GET)
Field name: Event type

DNS request fields

  • Event type
  • Domain
  • Status
  • Filtering rule
  • Start time
  • Elapsed time
  • Size

Tunnel request fields

  • Event type
  • Application
  • Domain
  • Remote Address
  • Status
  • Filtering rule
  • Start time
  • Elapsed time
  • Size
Member

ameshkov commented Feb 27, 2017

Log item details fields should depend on the event type. However, there are some common requirements.

Common requirements

  1. Tap on a field should automatically copy it's value to the clipboard. We should also show a toast about it.
  2. The Status field possible values: Processed, Blocked, Threat, Whitelisted, Redirect. The latter makes sense for Web request type only.
  3. Event type field possible values are Web request, DNS request, HTTPS tunnel
  4. Application field name depends on the app type and can be App, Browser, System app
  5. Filtering rule field is visible in two cases only: if the status is either Blocked or Whitelisted.
  6. Possible buttons in the item details footer: BLOCK, UNBLOCK (raised buttons), OPEN (flat button). OPEN button opens request URL in the default browser. OPEN is available for regular web requests with GET method only.
  7. Remote Address field format is 127.0.0.1:443 for IPv4 and [2001:4860:4860::8888]:443 for IPv6. Use IpAddressUtils.toString() method to get the proper format for it. Do not show this field if value is empty.

Regular web request fields

  • Event type
  • Application
  • Domain
  • Remote Address
  • Status (You should use Redirect status whenever redirectUrl field is not empty.)
  • Filtering rule
  • Request URL
  • Source URL (do not show if empty or null)
  • Redirect URL (show it if status is Redirect)
  • Content type (no need to translate, just use the enum value)
  • Start time (format: 18:45.123) (we don't really need date here)
  • Elapsed time (format: 123 ms)
  • Size (format: ↓123KB ↑5KB)

Please note, that Event type in the case of regular web request also contains HTTP method value.
It should look like:

Value: Web request (GET)
Field name: Event type

DNS request fields

  • Event type
  • Domain
  • Status
  • Filtering rule
  • Start time
  • Elapsed time
  • Size

Tunnel request fields

  • Event type
  • Application
  • Domain
  • Remote Address
  • Status
  • Filtering rule
  • Start time
  • Elapsed time
  • Size
@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Feb 28, 2017

Member

Filtering log menu item icon:
filtering_log_icons.zip

Fields icons:
filtering_log_fields_icons.zip

Member

ameshkov commented Feb 28, 2017

Filtering log menu item icon:
filtering_log_icons.zip

Fields icons:
filtering_log_fields_icons.zip

@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Feb 28, 2017

Member

@Revertron just caught a crash:

22:52:00.037 [main] ERROR c.a.a.c.CustomExceptionHandler - Thread: Thread[main,5,main]; Uncaught exception:

java.lang.UnsupportedOperationException: null
    at java.util.AbstractList.add(AbstractList.java:148) ~[na:0.0]
    at android.widget.ArrayAdapter.insert(ArrayAdapter.java:255) ~[na:0.0]
    at com.adguard.android.ui.FilteringLogActivity$3.run(FilteringLogActivity.java:119) ~[na:0.0]
    at android.os.Handler.handleCallback(Handler.java:751) ~[na:0.0]
    at android.os.Handler.dispatchMessage(Handler.java:95) ~[na:0.0]
    at android.os.Looper.loop(Looper.java:154) ~[na:0.0]
    at android.app.ActivityThread.main(ActivityThread.java:6119) ~[na:0.0]
    at java.lang.reflect.Method.invoke(Native Method) ~[na:0.0]
    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:886) ~[na:0.0]
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:776) ~[na:0.0]
Member

ameshkov commented Feb 28, 2017

@Revertron just caught a crash:

22:52:00.037 [main] ERROR c.a.a.c.CustomExceptionHandler - Thread: Thread[main,5,main]; Uncaught exception:

java.lang.UnsupportedOperationException: null
    at java.util.AbstractList.add(AbstractList.java:148) ~[na:0.0]
    at android.widget.ArrayAdapter.insert(ArrayAdapter.java:255) ~[na:0.0]
    at com.adguard.android.ui.FilteringLogActivity$3.run(FilteringLogActivity.java:119) ~[na:0.0]
    at android.os.Handler.handleCallback(Handler.java:751) ~[na:0.0]
    at android.os.Handler.dispatchMessage(Handler.java:95) ~[na:0.0]
    at android.os.Looper.loop(Looper.java:154) ~[na:0.0]
    at android.app.ActivityThread.main(ActivityThread.java:6119) ~[na:0.0]
    at java.lang.reflect.Method.invoke(Native Method) ~[na:0.0]
    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:886) ~[na:0.0]
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:776) ~[na:0.0]
@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Mar 1, 2017

Member

Rules constructor issues:

  1. Does not work properly for DNS requests
  2. Patterns should start with ||
  3. Add one more option: [x] Apply to the APPNAME requests only. In case if this box is checked, add $app modifier to the rule text.
Member

ameshkov commented Mar 1, 2017

Rules constructor issues:

  1. Does not work properly for DNS requests
  2. Patterns should start with ||
  3. Add one more option: [x] Apply to the APPNAME requests only. In case if this box is checked, add $app modifier to the rule text.
@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Mar 1, 2017

Member

I see that startappexchange.com SDK sometimes sends a GET request with URL length more than 20000. Actually, every time I start ru.sports app, I see this request. We'd better handle long URLs specifically.

Member

ameshkov commented Mar 1, 2017

I see that startappexchange.com SDK sometimes sends a GET request with URL length more than 20000. Actually, every time I start ru.sports app, I see this request. We'd better handle long URLs specifically.

@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Mar 1, 2017

Member

Add $important modifier support to the rules constructor dialog.

  1. Add a checkbox: Give a higher priority to the rule
  2. By default, this checkbox should be disabled.
  3. However, if you try to unblock a URL blocked with an $important rule, or block a whitelisted URL, make it checked.
Member

ameshkov commented Mar 1, 2017

Add $important modifier support to the rules constructor dialog.

  1. Add a checkbox: Give a higher priority to the rule
  2. By default, this checkbox should be disabled.
  3. However, if you try to unblock a URL blocked with an $important rule, or block a whitelisted URL, make it checked.
@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Mar 1, 2017

Member

Handle third-party flag in a smart way

  1. If web request is third-party, "Apply to third party requests only" option should be checked.
  2. Otherwise, let it be unchecked.
Member

ameshkov commented Mar 1, 2017

Handle third-party flag in a smart way

  1. If web request is third-party, "Apply to third party requests only" option should be checked.
  2. Otherwise, let it be unchecked.

@ameshkov ameshkov closed this Mar 2, 2017

@Revertron

This comment has been minimized.

Show comment
Hide comment
@Revertron

Revertron Mar 2, 2017

Member

Done.

Member

Revertron commented Mar 2, 2017

Done.

@Alex-302 Alex-302 reopened this Mar 2, 2017

@Alex-302

This comment has been minimized.

Show comment
Hide comment
@Alex-302

Alex-302 Mar 2, 2017

Member

Issues:

  1. Incorrect content type for Web-Socker connections
    wss://ws.hghit.com/ws
    Content type = Document
    Add @@||hdrezka.me^$content and go to hdrezka.me;
  2. After clearing, filtering log again contains the records;
  3. Request sorting is strange - requests from the page are located before request to the page [Moved to #1074]

image

  1. Elements, which was removed from HTML are not displayed(like in Adguard for Windows) [Moved to #1075]
Member

Alex-302 commented Mar 2, 2017

Issues:

  1. Incorrect content type for Web-Socker connections
    wss://ws.hghit.com/ws
    Content type = Document
    Add @@||hdrezka.me^$content and go to hdrezka.me;
  2. After clearing, filtering log again contains the records;
  3. Request sorting is strange - requests from the page are located before request to the page [Moved to #1074]

image

  1. Elements, which was removed from HTML are not displayed(like in Adguard for Windows) [Moved to #1075]
@ameshkov

This comment has been minimized.

Show comment
Hide comment
@ameshkov

ameshkov Mar 2, 2017

Member

Just fixed point 1.

Regarding point 3, this is by design (at least for now). We should fix it eventually, but we'd better move it to a separate issue.

Member

ameshkov commented Mar 2, 2017

Just fixed point 1.

Regarding point 3, this is by design (at least for now). We should fix it eventually, but we'd better move it to a separate issue.

@Alex-302

This comment has been minimized.

Show comment
Hide comment
@Alex-302

Alex-302 Mar 2, 2017

Member

Regarding point 3

done
#1074

Member

Alex-302 commented Mar 2, 2017

Regarding point 3

done
#1074

@Alex-302

This comment has been minimized.

Show comment
Hide comment
@Alex-302

Alex-302 Mar 2, 2017

Member

Added
4. Elements, which was removed from HTML are not displayed(like in Adguard for Windows)

Member

Alex-302 commented Mar 2, 2017

Added
4. Elements, which was removed from HTML are not displayed(like in Adguard for Windows)

@Revertron

This comment has been minimized.

Show comment
Hide comment
@Revertron

Revertron Mar 2, 2017

Member

Fixed point 2.

Member

Revertron commented Mar 2, 2017

Fixed point 2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment