Warning: Unable to determine whether or not callable is a generator with a return value [unable to parse decorated class methods]

[jefflester]

Description

If you have a decorated spider method, scrapy.utils.misc throws a warning saying that it cannot determine if the callable is a generator with a return value.

ast.parse() fails here when called with a decorated method.

Steps to Reproduce

I just copied the logic from misc.py and used it to analyze a class with the same overall code structure:

import re
import ast
import inspect

class Foo:
    @classmethod
    def func(self):
        """
        Description of func
        """  
        return

code = re.sub(r"^[\t ]+", "", inspect.getsource(Foo.func))
tree = ast.parse(code)

python3 test.py
> IndentationError: unexpected indent

The regex replacement isn't accounting for a possible decorator, so the code ends up looking like:

@classmethod
    def func(self):
        """
        Description of func
        """  
        return

Expected behavior: I'd like to be able to use decorated methods without dealing with noisy logs.

Actual behavior: My container logs are filled with tons of warning messages. The only workaround is to avoid the usage of decorators.

Reproduces how often: 100% of the time to my knowledge

Versions

$ scrapy version --verbose

Scrapy       : 2.5.1
lxml         : 4.6.4.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 21.7.0
Python       : 3.8.2 (default, Dec 21 2020, 15:06:04) - [Clang 12.0.0 (clang-1200.0.32.29)]
pyOpenSSL    : 21.0.0 (OpenSSL 1.1.1l  24 Aug 2021)
cryptography : 35.0.0
Platform     : macOS-10.15.7-x86_64-i386-64bit

Additional context

This is my first time filing a Scrapy issue. I'm happy to add more context if necessary, and apologies in advance if this has already been discussed elsewhere (fwiw I couldn't find anything).

[jpmckinney]

When this features was introduced in #3869, it used dedent, which would have avoided this issue. In #4935, this was changed to a simple regex, because apparently some people write code like:

class Bob:
    def doit(self):
        """
this line is flush left
        """
        if True:
            yield 1234

Anyway, to support both decorators and non-indented heredocs, then the regex needs to be made smarter.

[Godson-Gnanaraj]

import re
import ast
import inspect

class Foo:
    @classmethod
    def func(self):
        """
        Description of func
        """  
        return

    @classmethod
    def test(self):
        """
    Description of test
        """
        return


def parse(func):
    src = inspect.getsource(func)
    pattern = r"^[\t ]+"
    matches = re.findall(pattern, src)  # finds indentation
    code = re.sub(pattern, "", src)
    if matches:
        code = re.sub(f"\n{matches[0]}", "\n", code)  # remove indentation

    tree = ast.parse(code)

parse(Foo.func)
parse(Foo.test)
parse(parse)

Finding indentation of callable and removing it works.