Add AsyncSource with tests, docs and examples #207

szarykott · 2021-06-01T19:55:52Z

Resolves #148

This pull request adds a new type of source - AsyncSource, that supports async fn. Also, it modifies the builder in order to handle it properly. Builder implementation based on this article.

There is no implementation of such a source inside the library since there are various runtimes to choose from as explained in the rustdocs in this pull request.

One minor change - the type of Result in regular Source has been changed from type synonym defined inside the library to std's Result since ... type synonym was not exported from the crate.

Json5 tests have been modified for Windows. There was some issue with line endings I was not able to resolve in error message tests. CI for this package runs on Linux, so test will run.

matthiasbeyer

Hi!

Overall this looks really nice and I like that you added a few tests as well! I hope we can get this in!

I have a few things to remark, though! 😄

Cargo.toml

src/builder.rs

matthiasbeyer · 2021-06-03T07:14:02Z

src/source.rs

@@ -12,23 +14,65 @@ pub trait Source: Debug {

    /// Collect all configuration properties available from this source and return
    /// a HashMap.
-    fn collect(&self) -> Result<HashMap<String, Value>>;
+    fn collect(&self) -> Result<HashMap<String, Value>, ConfigError>;


Any particular reason why you don't use the crate::error::Result<_> anymore? If no, please revert this (and all similar) changes, so we don't have to specify ConfigError all the time in this module!

I saw that Result is not exported from the crate, which I remember vaguely did not play nicely with rust-analyzer that automatically used crate's method signature when asked to "implement trait functions". And since it wasn't exported, it didn't compile, I had to think about why. It was a long time ago, so it might be fixed now.

And personally, I am not a big fan of exporting Result, because in the case of import use config::* it overwrites std's Result and it is annoying. Hence the decision to change it here, since people have to implement it.

I am open to discussion though, it might be good idea to test it once more as well.

tests/file_json5.rs

szarykott · 2021-06-07T20:28:04Z

Thank you for the review! I'll try to fix things and answer in a few days.

szarykott · 2021-06-19T11:18:28Z

It seems there is a problem - due to const fn usage in reqwest dependency socket2 it fails on Rust 1.44.

I see few options:

try to figure out how to compile code that requires it only on new Rust
do not run CI on 1.44
try to play with dependencies to lower socket2 version transitively (requires to drop to tokio < 1 in examples)

Feel free to input your suggestions and in the meantime, I'll try to figure it out.

matthiasbeyer · 2021-06-19T11:23:50Z

Well, if we need a newer rust version because we need reqwest, the sensible thing would be to upgrade our MSRV to match that of reqwest.

From what I see, though, this crate is only required in the examples. I would therefore limit the requirement in the Cargo.toml so that it is only pulled in for the examples but not for the actual crate. If that is possible, we could exclude the examples from the 1.44 job and only compile them with stable rust.

What do you think?

matthiasbeyer · 2021-06-19T11:30:27Z

Try to cherry-pick matthiasbeyer@6f84f86 onto your branch if you like. This should fix the issue. (CI run here)

szarykott · 2021-06-22T19:33:49Z

Green. I dumped versions of packages that are only used in examples and tests to older versions that do not use fancy language features.

matthiasbeyer · 2021-06-22T19:57:11Z

Yeah, that change was unnecessary, please pull (fast-forward) from https://github.com/matthiasbeyer/config-rs/tree/pr-207 (or, if you don't mind, I can directly push my changes to this PR).

I also had a quick look at the commit history and I'm not happy yet, sorry. As soon as I'm happy with the overall changes and did a review, we should clean the history. There are a lot of unnecessary changes and commits in the history which should be removed. I can offer you help with that, even doing the rebase myself if you are okay with it / don't have the time / whatever!

Over all, though, I want to thank you very much for your patience and work on this crate! Async sources really is a feature we want to have here and I'm only so nit-picky here so we are doing the first steps right so we don't have to rewrite the feature later on (and that bit about the commit history is just my OCD on git history kicking in 😆 )! Please don't be offended by my nit-picking!

szarykott · 2021-06-26T09:59:55Z

@matthiasbeyer I'd be glad if you fixed the history in a way you see appropriate! Same goes for pushing the code you mentioned from your branch.

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

This patch adds the AsyncSource trait, the interface for providing async source functionality for this crate. Signed-off-by: Matthias Beyer <mail@beyermatthias.de> Reviewed-by: Matthias Beyer <mail@beyermatthias.de>

This patch rewrites the Config building mechanism using special objects for tracking the config building state. Transitions between states are done on the fly as required. This is required so that the async sources can be stored inside the configuration building objects, while keeping out the expenses in the non-async case, so a user of the crate has only to pay for what they are using (no async means no overhead for that). Signed-off-by: Matthias Beyer <mail@beyermatthias.de> Reviewed-by: Matthias Beyer <mail@beyermatthias.de>

matthiasbeyer · 2021-06-26T16:21:53Z

Feel free to review what I did and check whether I messed up somewhere... now this seems to be ready for merging (I'll do another review next week... too much to do today and in the next two days).

szarykott · 2021-06-29T17:20:12Z

It looks good to me

matthiasbeyer · 2021-06-29T17:59:12Z

Found some more things. Somehow the old tokio version slipped through my rebase, which is fixed now. Also, I simplified the example (also to showcase error conversion).

szarykott · 2021-07-03T18:24:14Z

Thank you. I pushed some commits to fix the pipeline.

Signed-off-by: Matthias Beyer <mail@beyermatthias.de> Reviewed-by: Matthias Beyer <mail@beyermatthias.de>

This is required because the examples pull in the "reqwest" crate, which depends on "socket2" and this crate fails to build if the "const fn" feature is not present (which wasn't on 1.44.0). Hence, we only run the tests in the 1.44.0 job, but do not compile the examples. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

Because one of our dependencies (namingly `socket2`) uses match in a const fn, which is stabilized in rust 1.46.0, we bump to this version as MSRV for this crate. Signed-off-by: Matthias Beyer <mail@beyermatthias.de> Tested-by: Matthias Beyer <mail@beyermatthias.de>

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

With this simplification, we save a bit of code on one side, but also showcase that errors from custom AsyncSource implementations are possible because the ConfigError type provides a variant for it. Signed-off-by: Matthias Beyer <mail@beyermatthias.de> Tested-by: Matthias Beyer <mail@beyermatthias.de>

matthiasbeyer · 2021-07-10T13:35:06Z

This PR has come a long way and I really, really appreciate the work you've put in here.
Thanks a lot!

I'm merging now! 🎉

matthiasbeyer requested changes Jun 3, 2021

View reviewed changes

szarykott force-pushed the async_source branch from 21a15fb to d455218 Compare June 19, 2021 11:07

Refactor: Extract setting of cache to helper fn

49f935a

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

matthiasbeyer force-pushed the async_source branch from 1bcb015 to 9922425 Compare June 26, 2021 14:00

matthiasbeyer and others added 4 commits June 26, 2021 17:30

Refactor: Import Result and use it instead of specifying full type path

ddc41bf

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

Add doc to fn

bbb61a6

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

Add AsyncSource trait

48e4a66

This patch adds the AsyncSource trait, the interface for providing async source functionality for this crate. Signed-off-by: Matthias Beyer <mail@beyermatthias.de> Reviewed-by: Matthias Beyer <mail@beyermatthias.de>

matthiasbeyer force-pushed the async_source branch 2 times, most recently from 9f2c585 to 8844f13 Compare June 26, 2021 16:10

szarykott force-pushed the async_source branch from 9cbb9bc to 2ae125e Compare July 3, 2021 18:18

szarykott and others added 7 commits July 3, 2021 20:30

Add example implementation using async source

f4de34e

Signed-off-by: Matthias Beyer <mail@beyermatthias.de> Reviewed-by: Matthias Beyer <mail@beyermatthias.de>

Add test for async builder

f8cef64

Signed-off-by: Matthias Beyer <mail@beyermatthias.de> Reviewed-by: Matthias Beyer <mail@beyermatthias.de>

Bump MSRV to 1.46.0

1a7ca3c

Because one of our dependencies (namingly `socket2`) uses match in a const fn, which is stabilized in rust 1.46.0, we bump to this version as MSRV for this crate. Signed-off-by: Matthias Beyer <mail@beyermatthias.de> Tested-by: Matthias Beyer <mail@beyermatthias.de>

Fix clippy: Use &str instead of &String as argument

55ba43d

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

Fix clippy: Use to_string() instead of clone()

18f01fd

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

szarykott force-pushed the async_source branch from 2ae125e to 33c6432 Compare July 3, 2021 18:31

matthiasbeyer merged commit 74a0a80 into mehcode:master Jul 10, 2021

szarykott deleted the async_source branch July 31, 2021 20:14

dlo9 mentioned this pull request Sep 28, 2021

Remove support for hjson #230

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AsyncSource with tests, docs and examples #207

Add AsyncSource with tests, docs and examples #207

szarykott commented Jun 1, 2021

matthiasbeyer left a comment

matthiasbeyer Jun 3, 2021

szarykott Jun 19, 2021

szarykott commented Jun 7, 2021

szarykott commented Jun 19, 2021

matthiasbeyer commented Jun 19, 2021

matthiasbeyer commented Jun 19, 2021

szarykott commented Jun 22, 2021

matthiasbeyer commented Jun 22, 2021 •

edited

szarykott commented Jun 26, 2021

matthiasbeyer commented Jun 26, 2021

szarykott commented Jun 29, 2021

matthiasbeyer commented Jun 29, 2021

szarykott commented Jul 3, 2021 •

edited

matthiasbeyer commented Jul 10, 2021

Add AsyncSource with tests, docs and examples #207

Add AsyncSource with tests, docs and examples #207

Conversation

szarykott commented Jun 1, 2021

matthiasbeyer left a comment

Choose a reason for hiding this comment

matthiasbeyer Jun 3, 2021

Choose a reason for hiding this comment

szarykott Jun 19, 2021

Choose a reason for hiding this comment

szarykott commented Jun 7, 2021

szarykott commented Jun 19, 2021

matthiasbeyer commented Jun 19, 2021

matthiasbeyer commented Jun 19, 2021

szarykott commented Jun 22, 2021

matthiasbeyer commented Jun 22, 2021 • edited

szarykott commented Jun 26, 2021

matthiasbeyer commented Jun 26, 2021

szarykott commented Jun 29, 2021

matthiasbeyer commented Jun 29, 2021

szarykott commented Jul 3, 2021 • edited

matthiasbeyer commented Jul 10, 2021

matthiasbeyer commented Jun 22, 2021 •

edited

szarykott commented Jul 3, 2021 •

edited