Improve file type detection using headers and ignoring URL arguments

[nvoxland-al]

🚀 🚀 Pull Request

Impact

• Bug fix (non-breaking change which fixes expected existing functionality)
• Enhancement/New feature (adds functionality without impacting existing logic)
• Breaking change (fix or feature that would cause existing functionality to change)

Description

Previously, we just looked at the end of the file string to determine the filetype in a ds.tensor_name.append(deeplake.read(...)) call.

For data read from URLs with query parameters, the end of the path was not necessarily the extension. Also, with URLs in general the path does not necessarily include the extension.

This PR removes any query parameters from the path when checking for a recognized extension, and also checks more header data to detect filetypes than the images we did previously.

Things to be aware of

Not all types can be detected by headers, but most are now

[sonarcloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
56.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud