Better endreport

[madeddy]

1. Atm we get two possible outcomes after unrpyc is done: Success or fail. The thing is skipped files are counted as "failed" and that's nonsense in my book. So i would like this changed.

   Counting this is so far done with booleans and this gives you just two possible answers, so we need another way. My idea is to write to a dict with three keys e.g: ok, fail, skip.

2. Unrpyc delivers no info about versions and users tend to omit a lot of stuff.
   I think we should simply include the current unrpyc version somewhere and print it at the end in some msg out. Along with the one from RenPys /renpy/vc_version.py. Maybe grab even the app name out of options.py and add it.

[CensoredUsername]

The thing is skipped files are counted as "failed" and that's nonsense in my book.

Fair, that is probably not the most wanted behavior, although for me it was often useful while working on it (because it's a reminder you probably meant to use --clobber.

Unrpyc delivers no info about versions and users tend to omit a lot of stuff.

Agreed. It'd be nice to have an option that just prints a lot of info for bug reports (python version, unrpyc version, file information). It'd be nice if we could get that from the rpyc file itself. Unfortunately that's a bit hard (funnily enough, rpyc files actually have a script_version field in there. But the value hasn't been changed in like 18 years.

I'll have to think about the nicest way to implement that for a bit.

[madeddy]

because it's a reminder you probably meant to use --clobber.

Sure. The same.
As said,i would just not count them as failed but as skipped. This means also we don't need to write this anymore for every skipped file out. One time at the end is enough e.g.
if skipped: "forgot --clobber, eh?"

info for bug reports (python version, unrpyc version, file information).

Oh right, i forgot: The header line should be cited if the fileheader not fits. We see instant what we got.

I'll have to think about the nicest way to implement that for a bit.

I would have volunteered to try my hand on all this, but i have no idea if this bangs with printlock, multiproc. etc. I thought also about some class for counting (shared state or so) file result. The same for all the fileinformation if not a namedtuple.

[madeddy]

OK. To get around the dependence on the return values for the outcome result of decompiling, i made a global dict for counting: result_count = {'total': 0, 'ok': 0, 'fail': 0, 'skip': 0, 'hdr_fail': 0}

I did try with different methods to write to it inside the multiprocessing loops, but it did not work to sync the counter values to the base dict.

• I put the counting inside with printlock statements.
• Added the dict even to the sharelock func.
• Did try to use mp.Manager().

Nothing. Any idea?

[CensoredUsername]

I did try with different methods to write to it inside the multiprocessing loops, but it did not work to sync the counter values to the base dict.

Due to the nature of mutliprocessing, each worker is actually executing in a different, completely separated, process. And thus they all have their own, completely separate globals in their memory space. Args and returned values are pickled and sent over a pipe internally, that's how python makes it work. If it were separate threads, then of course it wouldn't be that way. But due to the GIL using multiple threads in python to try to get better performance through parallelism is impossible. Maybe in 3.14 now that the GIL removal is proceeding in python dev.

What you want to do is just return the status as a result from the worker. Then afterwards in the main process you just process that list to figure out the total failures/successes.

I'd probably handle it like this.

# define special case exceptions for the special cases we're interesting in reporting differently.
# these should be raised by the relevant piece of code.
class AlreadyExists(Exception):
    pass

class BrokenHeader(Exception):
    pass

# return None on success. Returns the exception if something failed.
def worker(arg_tup):
    try:
        # code that tries to decompile a file here
        
        return None
    except Exception as e:
        # print exception like it currently does
        
        return e
        
# in main
    if args.processes > 1 and len(worklist) > 5:
        with Pool(args.processes, sharelock, [printlock]) as pool:
            results = pool.map(worker, worklist)
    else:
        results = list(map(worker, worklist))
        
    success = len(i for i in results if i is None)
    skipped = len(i for i in results if isinstance(i, AlreadyExists)
    failed = len(results) - success - skipped
    # etcetera

[madeddy]

What you describe and with extension on your system of return values i had already working. For the file skipping i (mis)used a raise ValueError.
In the end i didn't like it very much, as it did not really work for the translations and i see it as a bit restricted.
This is why i wanted to write the state wherever needed, which wasn't to this point to accomplish because of the sync issue.

Funnily i got the syncing working just before you answered above. 😵‍💫 Timing...
It uses now a global Manager() server for the counter-dict and(!) the lock. The dict is on different locations up-counted inside printlock. Interestingly the initializer/initarg isn't needed anymore, as the separate counting for the translation. Its now even flexible and extensible.

I upload it by chance, so you can test it if you want and add what you want. The multiprocessing "mock" will probably need to be looked after and I'm not sure i can do it. I see a change for Lock() and needed extension for Manager().