Skip to content
This repository

Cssfilter #477

Open
wants to merge 7 commits into from

5 participants

listen2 Neil Williams Jason Harvey cBlank
listen2

Updated to the latest cssutils, remove duplicate code, and a few misc improvements.

Neil Williams
Owner

Is there a change to setup.py that's missing?

r2/r2/lib/cssfilter.py
@@ -31,8 +31,8 @@
31 31
 from pylons.i18n import _
32 32
 from mako import filters
33 33
 
34  
-import os
35  
-import tempfile
  34
+#import os
1
Neil Williams Owner
spladug added a note July 10, 2012

Good find of the unused imports, but could you just delete the lines please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
r2/r2/lib/cssfilter.py
((72 lines not shown))
112 80
 }
113 81
 
  82
+cssutils.profile.addProfile("Reddit compat", custom_values, custom_macros)
3
Neil Williams Owner
spladug added a note July 10, 2012

Perhaps the profile name should be pulled out into a constant so it's not duplicated?

Nitpick: reddit should not be capitalized :)

listen2
listen2 added a note July 10, 2012

Can you explain what you mean by duplication?

Neil Williams Owner
spladug added a note July 10, 2012

The string "Reddit compat" is written out twice in this code. If you wanted to change the name it'd require changing in two places.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
r2/setup.py
@@ -82,7 +82,7 @@
82 82
         "cython>=0.14",
83 83
         "SQLAlchemy==0.7.4",
84 84
         "BeautifulSoup",
85  
-        "cssutils==0.9.5.1",
  85
+        "cssutils",
1
Neil Williams Owner
spladug added a note July 10, 2012

Looking at the changelog for cssutils, it looks like they break compatibility quite frequently. Would it be better to pin a specific (new) version?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Neil Williams
Owner

I'm running this new code on every stylesheet in the database currently and so far I've found a few regressions:

  • text-shadow does not appear to always work:
    • "black 0 0 10px" is not a valid value for CSS property "text-shadow"
    • "inherit" is not a valid value for CSS property "text-shadow"
  • Likewise for box-shadow:
    • "#0c0 0 0 10px" is not a valid value for CSS property "box-shadow"

I'll let you know if I come across more in my testing.

That said, this is already proving its worth as it's found several places where the subreddit stylesheets were flat out wrong! (some url(<a href="http://i.imgur.com/...">) type stuff and border: 1px solid 2px solid black), so this is very exciting. :)

Neil Williams
Owner

Additionally, and I'm not sure if this is new behaviour or not, I don't appear to get line numbers with my validation errors.

listen2

None of the three examples above are valid CSS. See the grammars at W3C [1] [2] [3]. The legacy code seems to be letting them through due to a custom rule. I can add that rule if you want to preserve the incorrect behavior.

It looks like the new cssutils doesn't report line numbers for validation errors, only for parse errors. If it's important, I can look again more closely later this week.

[1] http://www.w3.org/TR/css3-text/#text-shadow
[2] http://www.w3.org/TR/css3-background/#box-shadow
[3] http://www.w3.org/TR/css3-background/#ltshadowgt

Neil Williams
Owner

Both inherit and color length length length seem to validate just fine for me on the W3C CSS validator. MDN says that's valid syntax, though the CSS3 spec seems to indicate otherwise (but perhaps I'm misreading the grammar there).

listen2

Ah, my mistake; I misinterpreted the spec.

Neil Williams
Owner

Alrighty, looks like that covers everything in the existing stylesheets that is actually valid. Gonna do a final code-review pass then we should be good to go! :)

Jason Harvey
Owner
alienth commented May 09, 2013

Unfortunately there is an infinite recursion bug in cssutils 0.9.10 holding this back. We'll follow up once that can be addressed.

cBlank
cBlank commented May 09, 2013

Thanks a lot, can't wait. (Hopefully not another 10 months xD).

If it's still present, how can the infinite recursion bug be reproduced?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
171  r2/r2/lib/cssfilter.py
@@ -31,8 +31,6 @@
31 31
 from pylons.i18n import _
32 32
 from mako import filters
33 33
 
34  
-import os
35  
-import tempfile
36 34
 from r2.lib import s3cp
37 35
 
38 36
 from r2.lib.media import upload_media
@@ -45,9 +43,6 @@
45 43
 import cssutils
46 44
 from cssutils import CSSParser
47 45
 from cssutils.css import CSSStyleRule
48  
-from cssutils.css import CSSValue, CSSValueList
49  
-from cssutils.css import CSSPrimitiveValue
50  
-from cssutils.css import cssproperties
51 46
 from xml.dom import DOMException
52 47
 
53 48
 msgs = string_dict['css_validator_messages']
@@ -55,83 +50,55 @@
55 50
 browser_prefixes = ['o','moz','webkit','ms','khtml','apple','xv']
56 51
 
57 52
 custom_macros = {
58  
-    'num': r'[-]?\d+|[-]?\d*\.\d+',
59  
-    'percentage': r'{num}%',
60  
-    'length': r'0|{num}(em|ex|px|in|cm|mm|pt|pc)',
61  
-    'int': r'[-]?\d+',
62  
-    'w': r'\s*',
63  
-    
64  
-    # From: http://www.w3.org/TR/2008/WD-css3-color-20080721/#svg-color
65  
-    'x11color': r'aliceblue|antiquewhite|aqua|aquamarine|azure|beige|bisque|black|blanchedalmond|blue|blueviolet|brown|burlywood|cadetblue|chartreuse|chocolate|coral|cornflowerblue|cornsilk|crimson|cyan|darkblue|darkcyan|darkgoldenrod|darkgray|darkgreen|darkgrey|darkkhaki|darkmagenta|darkolivegreen|darkorange|darkorchid|darkred|darksalmon|darkseagreen|darkslateblue|darkslategray|darkslategrey|darkturquoise|darkviolet|deeppink|deepskyblue|dimgray|dimgrey|dodgerblue|firebrick|floralwhite|forestgreen|fuchsia|gainsboro|ghostwhite|gold|goldenrod|gray|green|greenyellow|grey|honeydew|hotpink|indianred|indigo|ivory|khaki|lavender|lavenderblush|lawngreen|lemonchiffon|lightblue|lightcoral|lightcyan|lightgoldenrodyellow|lightgray|lightgreen|lightgrey|lightpink|lightsalmon|lightseagreen|lightskyblue|lightslategray|lightslategrey|lightsteelblue|lightyellow|lime|limegreen|linen|magenta|maroon|mediumaquamarine|mediumblue|mediumorchid|mediumpurple|mediumseagreen|mediumslateblue|mediumspringgreen|mediumturquoise|mediumvioletred|midnightblue|mintcream|mistyrose|moccasin|navajowhite|navy|oldlace|olive|olivedrab|orange|orangered|orchid|palegoldenrod|palegreen|paleturquoise|palevioletred|papayawhip|peachpuff|peru|pink|plum|powderblue|purple|red|rosybrown|royalblue|saddlebrown|salmon|sandybrown|seagreen|seashell|sienna|silver|skyblue|slateblue|slategray|slategrey|snow|springgreen|steelblue|tan|teal|thistle|tomato|turquoise|violet|wheat|white|whitesmoke|yellow|yellowgreen',
66  
-    'csscolor': r'(maroon|red|orange|yellow|olive|purple|fuchsia|white|lime|green|navy|blue|aqua|teal|black|silver|gray|ActiveBorder|ActiveCaption|AppWorkspace|Background|ButtonFace|ButtonHighlight|ButtonShadow|ButtonText|CaptionText|GrayText|Highlight|HighlightText|InactiveBorder|InactiveCaption|InactiveCaptionText|InfoBackground|InfoText|Menu|MenuText|Scrollbar|ThreeDDarkShadow|ThreeDFace|ThreeDHighlight|ThreeDLightShadow|ThreeDShadow|Window|WindowFrame|WindowText)|#[0-9a-f]{3}|#[0-9a-f]{6}|rgb\({w}{int}{w},{w}{int}{w},{w}{int}{w}\)|rgb\({w}{num}%{w},{w}{num}%{w},{w}{num}%{w}\)',
67  
-    'color': '{x11color}|{csscolor}',
68  
-
69  
-    'bg-gradient': r'none|{color}|[a-z-]*-gradient\(.*\)',
  53
+    'bg-gradient': r'none|{color}|[a-z-]*-gradient\([^;]*\)',
70 54
     'bg-gradients': r'{bg-gradient}(?:,\s*{bg-gradient})*',
71 55
 
72  
-    'border-radius': r'(({length}|{percentage}){w}){1,2}',
73  
-    
74 56
     'single-text-shadow': r'({color}\s+)?{length}\s+{length}(\s+{length})?|{length}\s+{length}(\s+{length})?(\s+{color})?',
75  
-
76 57
     'box-shadow-pos': r'{length}\s+{length}(\s+{length})?(\s+{length})?',
77 58
 }
78 59
 
  60
+custom_macros = dict(   #re-use macros from the library
  61
+    custom_macros.items() +
  62
+    cssutils.profile._TOKEN_MACROS.items() +
  63
+    cssutils.profile._MACROS.items() +
  64
+    cssutils.profiles.macros[cssutils.profile.CSS3_BACKGROUNDS_AND_BORDERS].items() +
  65
+    cssutils.profiles.macros[cssutils.profile.CSS3_COLOR].items()
  66
+)
  67
+
79 68
 custom_values = {
80 69
     '_height': r'{length}|{percentage}|auto|inherit',
81 70
     '_width': r'{length}|{percentage}|auto|inherit',
82 71
     '_overflow': r'visible|hidden|scroll|auto|inherit',
83  
-    'color': r'{color}',
84  
-    'border-color': r'{color}',
85  
-    'opacity': r'^0?\.?[0-9]*|1\.0*|1|0',
  72
+
86 73
     'filter': r'alpha\(opacity={num}\)',
87 74
     
88 75
     'background': r'{bg-gradients}',
89 76
     'background-image': r'{bg-gradients}',
90  
-    'background-color': r'{color}',
91  
-    'background-position': r'(({percentage}|{length}){0,3})?\s*(top|center|left)?\s*(left|center|right)?',
92  
-    
93  
-    # http://www.w3.org/TR/css3-background/#border-top-right-radius
94  
-    'border-radius': r'{border-radius}',
95  
-    'border-top-right-radius': r'{border-radius}',
96  
-    'border-bottom-right-radius': r'{border-radius}',
97  
-    'border-bottom-left-radius': r'{border-radius}',
98  
-    'border-top-left-radius': r'{border-radius}',
99 77
 
100  
-    # old mozilla style (for compatibility with existing stylesheets)
101  
-    'border-radius-topright': r'{border-radius}',
102  
-    'border-radius-bottomright': r'{border-radius}',
103  
-    'border-radius-bottomleft': r'{border-radius}',
104  
-    'border-radius-topleft': r'{border-radius}',
105  
-    
106 78
     # http://www.w3.org/TR/css3-text/#text-shadow
107  
-    'text-shadow': r'none|({single-text-shadow}{w},{w})*{single-text-shadow}',
  79
+    'text-shadow': r'none|inherit|({single-text-shadow}{w},{w})*{single-text-shadow}',
108 80
     
109 81
     # http://www.w3.org/TR/css3-background/#the-box-shadow
110 82
     # (This description doesn't support multiple shadows)
111  
-    'box-shadow': 'none|(?:({box-shadow-pos}\s+)?{color}|({color}\s+?){box-shadow-pos})',
  83
+    'box-shadow': 'none|inherit|(?:({box-shadow-pos}\s+)?{color}|({color}\s+?){box-shadow-pos})',
  84
+    
  85
+    # old mozilla style (for compatibility with existing stylesheets)
  86
+    'border-radius-topright': r'{border-radius-part}',
  87
+    'border-radius-bottomright': r'{border-radius-part}',
  88
+    'border-radius-bottomleft': r'{border-radius-part}',
  89
+    'border-radius-topleft': r'{border-radius-part}',
112 90
 }
113 91
 
  92
+reddit_profile = "reddit compat"
  93
+cssutils.profile.addProfile(reddit_profile, custom_values, custom_macros)
  94
+cssutils.profile.defaultProfiles.append(reddit_profile)
  95
+
114 96
 def _build_regex_prefix(prefixes):
115 97
     return re.compile("|".join("^-"+p+"-" for p in prefixes))
116 98
 
117 99
 prefix_regex = _build_regex_prefix(browser_prefixes)
118 100
 
119  
-def _expand_macros(tokdict,macrodict):
120  
-    """ Expand macros in token dictionary """
121  
-    def macro_value(m):
122  
-        return '(?:%s)' % macrodict[m.groupdict()['macro']]
123  
-    for key, value in tokdict.items():
124  
-        while re.search(r'{[a-z][a-z0-9-]*}', value):
125  
-            value = re.sub(r'{(?P<macro>[a-z][a-z0-9-]*)}',
126  
-                           macro_value, value)
127  
-        tokdict[key] = value
128  
-    return tokdict
129  
-def _compile_regexes(tokdict):
130  
-    """ Compile all regular expressions into callable objects """
131  
-    for key, value in tokdict.items():
132  
-        tokdict[key] = re.compile('\A(?:%s)\Z' % value, re.I).match
133  
-    return tokdict
134  
-_compile_regexes(_expand_macros(custom_values,custom_macros))
  101
+cssutils.profile._compile_regexes(cssutils.profile._expand_macros(custom_values,custom_macros))
135 102
 
136 103
 class ValidationReport(object):
137 104
     def __init__(self, original_text=''):
@@ -190,9 +157,8 @@ def valid_url(prop,value,report):
190 157
      * image labels %%..%% for images uploaded on /about/stylesheet
191 158
      * urls with domains in g.allowed_css_linked_domains
192 159
     """
193  
-    try:
194  
-        url = value.getStringValue()
195  
-    except IndexError:
  160
+    url = value.uri
  161
+    if url == "":
196 162
         g.log.error("Problem validating [%r]" % value)
197 163
         raise
198 164
     # local urls are allowed
@@ -245,45 +211,10 @@ def valid_url(prop,value,report):
245 211
 
246 212
 
247 213
 def strip_browser_prefix(prop):
  214
+    if prop[0] != "-":
  215
+        return prop     #avoid regexp if we can
248 216
     t = prefix_regex.split(prop, maxsplit=1)
249  
-    return t[len(t) - 1]
250  
-
251  
-def valid_value(prop,value,report):
252  
-    prop_name = strip_browser_prefix(prop.name) # Remove browser-specific prefixes eg: -moz-border-radius becomes border-radius
253  
-    if not (value.valid and value.wellformed):
254  
-        if (value.wellformed
255  
-            and prop_name in cssproperties.cssvalues
256  
-            and cssproperties.cssvalues[prop_name](prop.value)):
257  
-            # it's actually valid. cssutils bug.
258  
-            pass
259  
-        elif (not value.valid
260  
-              and value.wellformed
261  
-              and prop_name in custom_values
262  
-              and custom_values[prop_name](prop.value)):
263  
-            # we're allowing it via our own custom validator
264  
-            value.valid = True
265  
-
266  
-            # see if this suddenly validates the entire property
267  
-            prop.valid = True
268  
-            prop.cssValue.valid = True
269  
-            if prop.cssValue.cssValueType == CSSValue.CSS_VALUE_LIST:
270  
-                for i in range(prop.cssValue.length):
271  
-                    if not prop.cssValue.item(i).valid:
272  
-                        prop.cssValue.valid = False
273  
-                        prop.valid = False
274  
-                        break
275  
-        elif not (prop_name in cssproperties.cssvalues or prop_name in custom_values):
276  
-            error = (msgs['invalid_property']
277  
-                     % dict(cssprop = prop.name))
278  
-            report.append(ValidationError(error,value))
279  
-        else:
280  
-            error = (msgs['invalid_val_for_prop']
281  
-                     % dict(cssvalue = value.cssText,
282  
-                            cssprop  = prop.name))
283  
-            report.append(ValidationError(error,value))
284  
-
285  
-    if value.primitiveType == CSSPrimitiveValue.CSS_URI:
286  
-        valid_url(prop,value,report)
  217
+    return t[1]
287 218
 
288 219
 error_message_extract_re = re.compile('.*\\[([0-9]+):[0-9]*:.*\\]\Z')
289 220
 only_whitespace          = re.compile('\A\s*\Z')
@@ -312,13 +243,13 @@ def validate_css(string):
312 243
         # directly, so we have to parse its error message string to
313 244
         # get it
314 245
         line = None
315  
-        line_match = error_message_extract_re.match(e.message)
  246
+        line_match = error_message_extract_re.match(e.args[0])
316 247
         if line_match:
317 248
             line = line_match.group(1)
318 249
             if line:
319 250
                 line = int(line)
320 251
         error_message=  (msgs['syntax_error']
321  
-                         % dict(syntaxerror = e.message))
  252
+                         % dict(syntaxerror = e.args[0]))
322 253
         report.append(ValidationError(error_message,e,line))
323 254
         return (None,report)
324 255
 
@@ -331,31 +262,23 @@ def validate_css(string):
331 262
             style = rule.style
332 263
             for prop in style.getProperties():
333 264
 
334  
-                if prop.cssValue.cssValueType == CSSValue.CSS_VALUE_LIST:
335  
-                    for i in range(prop.cssValue.length):
336  
-                        valid_value(prop,prop.cssValue.item(i),report)
337  
-                    if not (prop.cssValue.valid and prop.cssValue.wellformed):
338  
-                        report.append(ValidationError(msgs['invalid_property_list']
339  
-                                                      % dict(proplist = prop.cssText),
340  
-                                                      prop.cssValue))
341  
-                elif prop.cssValue.cssValueType == CSSValue.CSS_PRIMITIVE_VALUE:
342  
-                    valid_value(prop,prop.cssValue,report)
343  
-
344  
-                # cssutils bug: because valid values might be marked
345  
-                # as invalid, we can't trust cssutils to properly
346  
-                # label valid properties, so we're going to rely on
347  
-                # the value validation (which will fail if the
348  
-                # property is invalid anyway). If this bug is fixed,
349  
-                # we should uncomment this 'if'
350  
-
351  
-                # a property is not valid if any of its values are
352  
-                # invalid, or if it is itself invalid. To get the
353  
-                # best-quality error messages, we only report on
354  
-                # whether the property is valid after we've checked
355  
-                # the values
356  
-                #if not (prop.valid and prop.wellformed):
357  
-                #    report.append(ValidationError(_('invalid property'),prop))
358  
-            
  265
+                prop.name = strip_browser_prefix(prop.name)
  266
+                # check property name
  267
+                if not prop.name in cssutils.profile.propertiesByProfile(cssutils.profile.defaultProfiles): #TODO would populating an array at module init be faster?
  268
+                    report.append(ValidationError('invalid property',prop))
  269
+                    continue
  270
+
  271
+                # check property values
  272
+                # note that validateWithProfile can take a string with multiple values (eg "5px 10px"). No need to iterate.
  273
+                if not cssutils.profile.validateWithProfile(prop.name, prop.propertyValue.value)[0]:
  274
+                    error = (msgs['invalid_val_for_prop'] % dict(cssvalue = prop.propertyValue.cssText, cssprop = prop.name))
  275
+                    report.append(ValidationError(error, prop.propertyValue))
  276
+
  277
+                # Unlike above, we need to iterate over every value in the line
  278
+                for v in prop.propertyValue:
  279
+                    if v.type == cssutils.css.Value.URI:
  280
+                        valid_url(prop,v,report)
  281
+
359 282
         else:
360 283
             report.append(ValidationError(msgs['unknown_rule_type']
361 284
                                           % dict(ruletype = rule.cssText),
2  r2/setup.py
@@ -82,7 +82,7 @@
82 82
         "cython>=0.14",
83 83
         "SQLAlchemy==0.7.4",
84 84
         "BeautifulSoup",
85  
-        "cssutils==0.9.5.1",
  85
+        "cssutils==0.9.10b1",
86 86
         "chardet",
87 87
         "psycopg2",
88 88
         "pycountry",
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.