[💡 Feature]: shared-store atomic operations

[pedrorfernandes]

Is your feature request related to a problem?

I have a test scenario where I need to access specific url's, e.g.: ['url01.com', 'url02.com', 'url03.com']

The wdio workers should share these URL's and never clash. Once a worker is finished, it "frees" up that url for another worker.

Possibly a better way to handle this is through network/load balancing, however I wanted to try this with shared-data service, setting up the url sharing between workers:

// worker A
await getValue("*"); // { 'url01.com': true, 'url02.com': true } booleans represent isUrlAvailable

// worker B
await getValue("*"); // { 'url01.com': true, 'url02.com': true }

// worker A chooses `url01.com`
await setValue("url01.com", false)
// worker B chooses `url01.com`
await setValue("url01.com", false)

// both A + B workers now have the same URL
// clash !

This is basically a shared queue management problem between node worker threads

Describe the solution you'd like.

I could implement in shared-store another server operation '/compareAndSet' to avoid collisions when writing data

compareAndSet(key, expectedValue, newValue) only sets newValue if expectedValue is the current value in the store

// worker A
await getValue("*"); // { 'url01.com': true, 'url02.com': true }

// worker B
await getValue("*"); // { 'url01.com': true, 'url02.com': true }

// worker A chooses `url01.com`
await compareAndSet("url01.com", true, false)
// worker B chooses `url01.com`
await compareAndSet("url01.com", true, false) // throws error, 'url01.com' is not `=== true` anymore
// worker B must retry another value

Describe alternatives you've considered.

I thought about a different service where we just configure a queue of resources to be shared [1, 2, 3, 4] and the operations would just be /take a resource and /return a resource, this is easier on the beforeSession for each worker since the server would automatically dispense the value the worker needs and it doesn't need to retry until a value is free

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[christian-bromann]

Hey @pedrorfernandes, thanks for filing the issue.

This seems indeed like useful idea. We would appreciate any help on this so please help us out! Have a look at our contributing guidelines and reach out if you have any questions. Cheers!

[Ajneesh21]

Hey looks intresting and I would like to try this out if I am assigned this issue.

[christian-bromann]

Awesome @Ajneesh21 , let me know if you have any question contributing to the project. Note, we also have office hours to connect to contributors.

[christian-bromann]

@pedrorfernandes I wonder what would you suggest to provide an easier workflow going through multiple values. According to your proposal the compareAndSet would fail if the expected value is not met. Given your example a user would have to run a for loop to iterate over 4 possible values. I think it is worth thinking about another API that allows to set a pool of values and allow each worker to claim and release values from it. For example:

const url = await getValueFromPool('hostnames') // one of ['url01.com', 'url02.com', 'url03.com']

then after the test:

await releaseValueFromPool('hostnames', url)

What do you think?

[pedrorfernandes]

Yes @christian-bromann that's similar to what I described in the Describe alternatives you've considered.

I think the resource queue api would be easier to manage on the workers onBeforeSession() since we don't need to catch errors and retry. The set and compare option can be used for other purposes rather than a specific resource queue.

Other than those 2 API calls I would also need a way to configure:

Possibly on the service setup:

export const config = {
    // ...
    services: [
        ['shared-store', {
            resourcePools: {
                 hostnames: ['url01.com', 'url02.com', 'url03.com'] 
                 // issue: if these values were fetched asynchronously, 
                 // then it can't be done this way in the config file (sync)
            }
        }]
    ],
    // ...
}

Or on prepare:

onPrepare() {
   // these values can be fetched asynchronously
   await setResourcePool('hostnames', ['url01.com', 'url02.com', 'url03.com']);
}

And then usage

beforeSession() {
   const url = await getValueFromPool('hostnames') // one of ['url01.com', 'url02.com', 'url03.com']
}

afterSession() {
   await releaseValueFromPool('hostnames', url)
}

[christian-bromann]

@Ajneesh21 note that @pedrorfernandes took the initiative to implement this. I am happy to assign a new task to you after you went through the workshop.

[christian-bromann]

Let's continue convo in #10029, thanks for taking a stab at this @pedrorfernandes