getting started with high level API for the library

[akshaysharmajs]

@Gallaecio @elacuesta Please discuss here the basic input-output structure of the API.

Can you please mention the issues you were talking about related to API in the main scrapy repo?

[Gallaecio]

I would like you to determine, based on the MIME-sniffing specification, what data you need as input.

So, for example, let’s say that we start with a basic API like:

def get_mime(*, body: bytes, content_type_headers: list): -> str
    pass

This API most likely needs to be extended to be able to support all inputs that affect the MIME-sniffing algorithm. For example, you probably need an additional, boolean parameter for the no-sniff flag (which could eventually be exposed to Scrapy users as a setting).

So, what other input data is needed?

[akshaysharmajs]

What I read in the MIME-sniffing specification and algorithms, we need these as input:

• body
• supplied MIME-type(Content-Type)
• check-for-apache-bug flag (related to some bug associated with Apache for text or binary files)
• no-sniff flag (Ofcourse)
• sniff-scriptable flag (for scriptable mime types like XML mimetype, HTML mimetype)

[Gallaecio]

Does the check-for-apache-bug flag really need to be specified an input? It seems to be used only internally.

On the other hand, reading 5.1, I see the algorithm changes depending on the origin of the resource: if it’s HTTP, the check-for-apache-bug flag is used, otherwise (file system or some other protocol) it is not. So we probably need an input parameter for this. We could have a protocol parameter, so that the input value is the protocol as seen in the URL (Scrapy supports http, https, ftp, file, data, and s3 by default), and internally we could use the HTTP logic if the value is http or https and the other logic otherwise. I suggest this instead of a simpler parameter to make it easier to provide the input parameter (urlparse(url).scheme) and so that we do not need API changes if the specification changes behavior for some protocol, but I do not have a strong opinion about it, we could also simplify this as a boolean parameter (e.g. http_origin: boolean = True).

Regarding the sniff-scriptable flag, reading 7 I believe it is also only used internally. Its value is the opposite of the no-sniff flag, so the no-sniff input parameter should suffice.

In 7.1, step 1.2, you can read:

User agents may implicitly extend this table to support additional MIME types.

However, user agents should not implicitly extend this table to include additional byte patterns for any computed MIME type already present in this table, as doing so could introduce privilege escalation vulnerabilities.

I think this means we should allow for a table of additional rows as input, and we should probably raise a ValueError exception if the input table does not conform to that second paragraph.

[akshaysharmajs]

For starting with API, I am using (Body, Content-Type header, boolean parameter for HTTP, boolean no-sniff flag) as Input and simple mimetype as output.

And maybe create another function for determining the resource metadata by following the algorithm mentioned in section 5.1.?

[Gallaecio]

For starting with API, I am using (Body, Content-Type header, boolean parameter for HTTP, boolean no-sniff flag) as Input and simple mimetype as output.

👍

And maybe create another function for determining the resource metadata by following the algorithm mentioned in section 5.1.?

I expect the main function to internally call other functions, so that the code remains readable. But I would keep those other functions private (prefixed with _), at least for now, so that we can freely change their API in the future without concern for backward compatibility. We can always make some of them public in the future if there’s demand for it from users.