`no_std` support for the `url` crate #831

domenukk · 2023-04-08T15:32:34Z

This extends/rebases/fixes #717.
All checks seem to pass.
For no_std support, however, as mentioned in #717, this still needs a bunch of fixes. Specifically, It looks like we'll have to use the ip_in_core nightly feature, or use this crate: https://docs.rs/no-std-net. I'd personally opt for nightly, as it looks like it will be merged eventually, but I could do both.

/edit: For future reference, in the meantime, net and the Error trait have made their way into core

Add default feature flag "std" that enables a `std::error::Error` impl for `Errors`.

domenukk · 2023-04-08T16:09:12Z

Went for ip_in_core now, so it needs nightly, but seems to work.

url/src/lib.rs

domenukk · 2023-04-10T13:54:26Z

I spent some time trying to get serde no_std support working, but it doesn't play nicely with the ip_in_core nightly feature - it throws errors like

the trait `Serialize` is not implemented for `Ipv6Addr`

valenting · 2023-04-10T13:59:54Z

Went for ip_in_core now, so it needs nightly, but seems to work.

If it breaks stable, that's obviously a no-go.
Also, our msrv is 1.56.0, so preferably the no_std changes would also work on that version.

codecov · 2023-04-10T13:59:55Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (main@54346fa). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #831   +/-   ##
=======================================
  Coverage        ?   81.82%           
=======================================
  Files           ?       21           
  Lines           ?     3549           
  Branches        ?        0           
=======================================
  Hits            ?     2904           
  Misses          ?      645           
  Partials        ?        0

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

domenukk · 2023-04-10T14:15:12Z

It should not break stable, it should only "break" no_std (which never worked in the first place).
Switching to no-std-net also doesn't fix the problem with serde - the project seems stale/would need this to be merged:
dunmatt/no-std-net#16

domenukk · 2023-04-10T14:32:05Z

Moving to no-std-net anyway, since it seems to fit your msrv better

domenukk · 2023-04-16T23:43:04Z

Since I do need the nightly version, but I can see why some people wouldn't, I added two features:

no_std_net uses the no_std_net crate, no nightly needed (but the extra dependency)
unstable enabled unstable net_in_core feature, and also the error_in_core feature. Due to its opt-in nature, I would assume this is an okay solution (?)

Else we can wait until the features get stabilised.
Feedback welcome.

domenukk · 2023-04-16T23:45:21Z

idna/src/uts46.rs

@@ -453,7 +453,7 @@ impl Idna {
            return Errors::default();
        }
        let mut errors = processing(domain, self.config, &mut self.normalized, out);
-        self.output = std::mem::replace(out, String::with_capacity(out.len()));
+        self.output = core::mem::replace(out, String::with_capacity(out.len()));


FYI, this fixes a bug in the current idna no_std support. Why is it not caught in CI, and should I open an extra PR for it?

I recently dabbled with no_std stuff as well and I think the sure-fire way to test no_std compatibility is to:

target a platform that does not have std (certain ARM platforms) OR

have a really simple no_std binary crate which uses the libraries

Yes I'm trying it with cargo +nightly build -Zbuild-std=core,alloc --target aarch64-unknown-none -v --release --no-default-features

Could you split this off into a separate PR?

…ml on older rust versions)

domenukk · 2023-05-31T13:01:33Z

Any news on this? :)

domenukk · 2023-11-23T13:29:52Z

Any hint on what to do with codecov CI?

madsmtm

I think the guiding principle for this feature should be: "Don't inconvenience people who don't care about no_std", which is... All of url's current users ;)

So, I really don't think we should bump MSRV for this feature - but a way to solve that would be to use rustc_version in a build script to detect when the feature is available. (This does come with a slight compile-time cost, but hopefully that's acceptable, and it will go away once the functionality is in url's MSRV).

madsmtm · 2023-12-22T06:22:11Z

url/Cargo.toml


 [features]
-default = []
+default = ["std"]


I fear this might be a fairly big breaking change?

When I added no_std support for the other libraries in this crate, it was more trivial, since there was only three crates that disabled default features - but that is not the case for url; there are many crates specifying default-features = false, so even though there was no default features previously, this change will affect all of those crates.

Python script for finding crates using `default-features = false`.

import json import requests import sys API_URL = "https://crates.io/api/v1/crates" crate_name = sys.argv[1] page = 1 while True: # print(f"page {page}") data = json.loads(requests.get(f"{API_URL}/{crate_name}/reverse_dependencies?per_page=100&page={page}").content) page += 1 if len(data['versions']) == 0: break versions = {x['id']: x for x in data['versions']} for dep in data['dependencies']: if not dep['default_features']: crate = versions[dep['version_id']] print(f"{crate['crate']} had `default-features = false`")

I don't see any other way?

Well... I see two:

Bundling this in with other breaking changes, and releasing url v0.3

Calling the feature unstable-no-std instead, to signify that it may break other crates in the dependency chain, and that it is a bad way of doing it which will be rectified once v0.3 is released.

madsmtm · 2023-12-22T06:27:23Z

url/Cargo.toml

+no_std_net = ["no-std-net"]
+# UNSTABLE FEATURES (requires Rust nightly)
+# For no_std: Use errors_in_core and net_in_core
+unstable = ["idna/unstable"]


rust-lang/rust#108443 is FCP-ing now, so it'll probably land in a few months - perhaps we should wait with this feature until then?

Right now this PR also uses error_in_core (rust-lang/rust#103765) but that part can be removed if we want to go for stable instead

The value in that feature is that this could become a non-breaking change (other than bumping MSRV, which is probably acceptable).

Update: core::net has landed in Rust 1.77.0

url/Cargo.toml

mspiegel · 2024-07-16T21:21:42Z

Is this PR still maintained? no_std support can now be achieved using stable rust 1.81. I have a patch ready that keeps the MSRV at 1.56 for the feature std. I see from the comments this would be released as a breaking change for the url crate.

domenukk · 2024-07-16T21:54:53Z

I'm happy to help to get this patch, your patch, or a mix of both into the url crate, but of course up to the maintainers regarding pace.
I think keeping the MSRV for std, while adding no_std support for the latest stable rust could be a good way forward? (If that is possible at all)

Manishearth · 2024-07-16T22:08:33Z

So the upcoming changes to URL affect its msrv but not that intensely, otherwise I'd suggest piggybacking on that.

I'm mostly worried about --no-default-features users having an MSRV change. Do we have an idea as to how many of those there are?

Would be nice to land though.

mspiegel · 2024-07-17T00:08:00Z

I modified the Python script (above) for finding crates using default-features = false. The crates are printed along with the version requirement string for the url dependency. All the crates are using some variation of ^2. I was relieved to see nothing with a > comparison requirement. There appears to be a consensus that changing the default features of the url crate would increment the major version number. I have opened #953 for discussion. The PR includes a new section in the crate documentation that explains the breaking change from version 2 to version 3 when default-features = false.

Manishearth · 2024-07-17T01:24:17Z

There appears to be a consensus that changing the default features of the url crate would increment the major version number.

Wait, where is this consensus? This is not my experience with Rust.

Manishearth · 2024-07-17T01:27:38Z

To be clear, my worry is specifically that if this feature is implemented in a way that changes our MSRV in --no-default-features mode, and we have users in --no-default-features mode that care about MSRV, then we should be a bit wary. But I suspect we do not have such users given that the crate currently doesn't have any default features.

And even if we did, I wouldn't consider it breaking, I would consider it a "tread carefully".

domenukk · 2024-07-17T01:33:21Z

url/src/lib.rs

@@ -73,6 +73,14 @@ assert!(data_url.fragment() == Some(""));
 # run().unwrap();
 ```

+## Default Features


@mspiegel I've merged your PR into mine here

madsmtm · 2024-07-17T01:36:50Z

I agree that the SemVer breakage is lessened now that both Error and the net types are available in core, but even if we're fine with bumping MSRV for users of default-features = false, the breakage is not eliminated, since we're still making certain APIs unavailable to those users.

E.g. both this PR and #953 will still break users of default-features = false if they also use Url::socket_addrs, Url::from_file_path, Url::from_directory_path or Url::to_file_path.

Co-authored-by: Michael Spiegel <michael@hushmesh.com>

domenukk · 2024-07-17T01:45:40Z

url/src/lib.rs

@@ -73,6 +73,14 @@ assert!(data_url.fragment() == Some(""));
 # run().unwrap();
 ```

+## Default Features
+
+Versions `< 3` of the crate have no default features. Versions `>= 3` have the default feature 'std'.


This will only be true if we really need to change version numbers

I realized that changing the text to say <= 2.5.2 and > 2.5.2 so that the docs are correct regardless of whether it’s a major or minor bump.

domenukk · 2024-07-17T01:59:53Z

FWIW just noticed that Error is going to be available in 8 days from now with 1.80.0, so don't merge just yet! :)

(Else we can also leave out the Error trait for now and add it later, but it'll be a new MSRV discussion)

madsmtm and others added 8 commits February 17, 2022 13:58

Make form_urlencoded no_std compatible

35653b0

Make data-url no_std compatible

194122c

Make idna no_std compatible

422a260

Add default feature flag "std" that enables a `std::error::Error` impl for `Errors`.

Test no_std support

81ee074

Initial work at making url no_std compatible

67d2a4c

Merge branch 'master' into no_std

db92dd2

Fixed testcases for no_std

bb7955a

fix no_std for nightly

a98aef8

domenukk marked this pull request as ready for review April 8, 2023 16:09

Add no_std to CI

0d14d5a

domenukk commented Apr 8, 2023

View reviewed changes

url/src/lib.rs Outdated Show resolved Hide resolved

Fix no_std for deps

315eb43

Moved to no-std-net

6d622ab

domenukk force-pushed the no_std branch from a2093b6 to 6d622ab Compare April 10, 2023 14:41

Added unstable, no_std_net options

3efe254

domenukk commented Apr 16, 2023

View reviewed changes

OverOrion mentioned this pull request Apr 18, 2023

Add support for no_std integritee-network/rust-url#1

Merged

domenukk added 4 commits April 27, 2023 15:28

Make CI happy by removing no-std-net/serde (not supported in Cargo.to…

36bc5b3

…ml on older rust versions)

Merge branch 'master' into no_std

29bfcf7

Only test aarch64 on ubuntu

e16eb2b

Fix CI some more

3dd87d8

OverOrion mentioned this pull request May 12, 2023

No std upstream working rust-nostr/nostr#90

Closed

3 tasks

Merge branch 'master' into no_std

316c868

domenukk mentioned this pull request Dec 5, 2023

no-std support for hickory-proto hickory-dns/hickory-dns#2104

Draft

madsmtm mentioned this pull request Dec 22, 2023

no_std support for all crates #717

Closed

madsmtm reviewed Dec 22, 2023

View reviewed changes

yukibtc mentioned this pull request Jan 21, 2024

Remove url-fork dep rust-nostr/nostr#258

Open

Kailai-Wang mentioned this pull request Mar 22, 2024

Prefer no-std url crate litentry/litentry-parachain#2611

Merged

no_std support for url

43c7d41

madsmtm mentioned this pull request Jul 16, 2024

no_std support? #609

Open

mspiegel added 2 commits July 16, 2024 17:37

apply cargo fmt

74c0910

bugfix for 'std' and windows platform interaction

a544730

Explanation of default features change.

619de8e

mspiegel mentioned this pull request Jul 16, 2024

no_std support for the url crate with no breaking changes for feature std #953

Closed

domenukk added 2 commits July 17, 2024 02:58

Merge branch 'main' into no_std

00efb8f

Merge remote-tracking branch 'mspiegel/no_std' into no_std

fc0778f

Remove no-std-net

027e1c7

domenukk added 2 commits July 17, 2024 03:29

fix CI

d367346

Even more CI

7377a57

domenukk commented Jul 17, 2024

View reviewed changes

Add mspielgel as co-author

d43506b

Co-authored-by: Michael Spiegel <michael@hushmesh.com>

domenukk commented Jul 17, 2024

View reviewed changes

Cleanup Errors in tests

c8d8e6d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`no_std` support for the `url` crate #831

`no_std` support for the `url` crate #831

domenukk commented Apr 8, 2023 •

edited

Loading

domenukk commented Apr 8, 2023

domenukk commented Apr 10, 2023

valenting commented Apr 10, 2023

codecov bot commented Apr 10, 2023 •

edited

Loading

domenukk commented Apr 10, 2023

domenukk commented Apr 10, 2023

domenukk commented Apr 16, 2023

domenukk Apr 16, 2023

OverOrion Apr 18, 2023

domenukk Apr 27, 2023

valenting Jun 14, 2023

domenukk commented May 31, 2023

domenukk commented Nov 23, 2023

madsmtm left a comment

madsmtm Dec 22, 2023

domenukk Dec 22, 2023

madsmtm Dec 22, 2023

madsmtm Dec 22, 2023

domenukk Dec 22, 2023

madsmtm Dec 22, 2023

madsmtm Mar 21, 2024

mspiegel commented Jul 16, 2024

domenukk commented Jul 16, 2024 •

edited

Loading

Manishearth commented Jul 16, 2024

mspiegel commented Jul 17, 2024

Manishearth commented Jul 17, 2024

Manishearth commented Jul 17, 2024

domenukk Jul 17, 2024

mspiegel Jul 17, 2024

madsmtm commented Jul 17, 2024 •

edited

Loading

domenukk Jul 17, 2024

mspiegel Jul 17, 2024

domenukk commented Jul 17, 2024 •

edited

Loading

no_std support for the url crate #831

Are you sure you want to change the base?

no_std support for the url crate #831

Conversation

domenukk commented Apr 8, 2023 • edited Loading

domenukk commented Apr 8, 2023

domenukk commented Apr 10, 2023

valenting commented Apr 10, 2023

codecov bot commented Apr 10, 2023 • edited Loading

Codecov Report

domenukk commented Apr 10, 2023

domenukk commented Apr 10, 2023

domenukk commented Apr 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domenukk commented May 31, 2023

domenukk commented Nov 23, 2023

madsmtm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mspiegel commented Jul 16, 2024

domenukk commented Jul 16, 2024 • edited Loading

Manishearth commented Jul 16, 2024

mspiegel commented Jul 17, 2024

Manishearth commented Jul 17, 2024

Manishearth commented Jul 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

madsmtm commented Jul 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domenukk commented Jul 17, 2024 • edited Loading

`no_std` support for the `url` crate #831

`no_std` support for the `url` crate #831

domenukk commented Apr 8, 2023 •

edited

Loading

codecov bot commented Apr 10, 2023 •

edited

Loading

domenukk commented Jul 16, 2024 •

edited

Loading

madsmtm commented Jul 17, 2024 •

edited

Loading

domenukk commented Jul 17, 2024 •

edited

Loading