Feat: Concurrent manifest execution

[JustinBacher]

I'm submitting a

• bug fix
• feature
• documentation addition

What is the current behavior?

Comtrya currently executes manifests sequentially, running each one in order of their dependencies. This leads to slower execution times when manifests could safely run concurrently.

What is the expected behavior?

Manifests should execute concurrently when they have no dependents.

What is the motivation / use case for changing the behavior?

Faster manifest execution using concurrent processing where possible.

Details:

• Restructured apply command.
  • Moved DAG to it's own module named dependency_manager
  • Used barriers to stall manifests until dependencies complete
  • Moved manifest exec logic to the manifest struct itself
• Added single password prompt at beginning to handle child password prompts
  • Since child password prompts get lost in the output it's unclear that a child is waiting on input
• Added dependencies:
  • rpassword
  • zeroize
  • async-trait

Known Issues / TODO:

• Need review of any functions still needing to be made async
• Looking for testing assistance for functionality verification
• Exec atom currently hangs during execution
  • Currently reading child stdin/stdout eternally
  • Still sends child password prompts to terminal (shouldn't since i have it ask for password up front)

Please tell us about your environment:

Version (comtrya --version): v0.9.1
Operating system: CachyOS

[martintc]

This is a problem since the implementation in password_manager.rs is for linux only. Is this due to just testing and development on linux or can this functionality not work on windows/macOS/BSD(s)?

[JustinBacher]

Due to testing on linux only. I'll need to first get linux working then I can work on windows/macOS/BSD. I just don't know how I'm gonna get it to stop asking for password while running elevated commands.

[JustinBacher]

So the issue with it asking for password during any package manager operations even though I'm using the PasswordManager is here. Both Paru and Yay use sudo under the hood and since sudo defaults to /dev/tty, it's overriding the piped stdin in exec.

That said the only fix I have been able to come up with (that's worked so far) is this:

...
atom: Box::new(Exec {
    command: String::from("echo"),
    arguments: [
            command: String::from(secret),
            command: String::from("|"),
            command: String::from("paru"),
            ...

I tried immediately writing the secret in exec instead of echoing like this but for some reason sudo isn't picking it up.

This may need similar workarounds for other package managers unless someone can come up with a better way of doing this

[martintc]

Pulled the branch to run it. With just some initial running, I came across this being spit out in the comtrya output.

thread 'tokio-runtime-worker' panicked at /Users/toddmartin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/blocking/shutdown.rs:51:21:
Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at /Users/toddmartin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/blocking/shutdown.rs:51:21:
Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.
thread 'tokio-runtime-worker' panicked at /Users/toddmartin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.43.0/src/runtime/blocking/shutdown.rs:51:21:
Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.

This looks like it may be a good reference.
https://stackoverflow.com/questions/65426683/why-does-tokio-return-the-error-cannot-drop-a-runtime-in-a-context-where-blocki

This occurs when running

cargo run --bin comtrya -- -vv -d examples/file apply -m file-chown,file-download-chown,file-download,file-unarchive,fileremove,walk

[JustinBacher]

Pulled the branch to run it. With just some initial running, I came across this being spit out in the comtrya output.

Just saw this, without testing; it's likely just because of the use of blocking calls from the main branch, I didn't test file downloads or git clones when I was developing. They weren't on my radar. Looks like here and above a couple lines in reqwest both use the blocking protocol and that won't work with async. Probably just changing these will fix it, I'll take a look and correct when I get a chance.

[JustinBacher]

gix works with async_std not regular async