Fix: local path issue in distributed optimize method

[deependujha]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Fixes #193

To test it, run the script:

import os
from litdata import optimize

def compress(index):
    return (index, index ** 2)

optimize(
    fn=compress,
    inputs=list(range(100)),
    num_workers=2,
    output_dir="my_output",
    chunk_bytes="64MB",
    num_nodes=2,
)

The optimized dataset will be present in: /teamspace/jobs/{job_name}/nodes-0/my_output.

Read the optimized data using:

from litdata import StreamingDataset

output_dir = "/teamspace/jobs/litdata-optimize-2024-07-08-17-08-51/nodes.0/my_output"

dataset = StreamingDataset(output_dir)

print(dataset[:])

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

Attention: Patch coverage is 42.85714% with 4 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@14a47f2). Learn more about missing BASE report.

Additional details and impacted files

@@          Coverage Diff          @@
##             main   #214   +/-   ##
=====================================
  Coverage        ?    77%           
=====================================
  Files           ?     33           
  Lines           ?   4709           
  Branches        ?      0           
=====================================
  Hits            ?   3626           
  Misses          ?   1083           
  Partials        ?      0           

[tchaton]

@deependujha Looks great ! Can you add an example in the README in another PR ?