Drop MAX_STATEMENTS in write use_ti logic in versions > 11.8.15

[MariusWirtz]

I was just informed that TM1 release 11.8.15 will not have a limitation on the number of statements that a TI process can have.

We can increase the MAX_STATEMENTS logic in the write_through_unbound_process function. This should lead to better performance as the CellPut statements won't need to be broken down into n sequentially executed processes.

[MariusWirtz]

Implemented in #808

While this feature certainly helps in cloud environments, the performance benefit of using larger chunks is minimal when running on localhost.

The bottleneck actually lies in the creation of the process. When using normal processes and breaking it down into two separate steps it is transparent to see that the creation/registration of the process is the actual bottleneck.

For a TI with 216k lines in the prolog it takes about 9seconds to be created and only 2 seconds to be executed.
Using an unbound process takes ~11 seconds.

Perhaps this is something we can raise with IBM? Perhaps IBM can accelerate the creation of unbound processes.

[MariusWirtz]

Here is a sample reproducing the issue

from TM1py import TM1Service, Dimension, Hierarchy, Cube, Process

CUBE_NAME = "TM1py Cube"
DIMENSION_NAMES = "TM1py Demension 1", "TM1py Dimension 2"
PROCESS_NAME = "TM1py Process"

CELL_UPDATES = 250_000


def prep(tm1):
    hierarchy = Hierarchy(DIMENSION_NAMES[0], DIMENSION_NAMES[0])
    hierarchy.add_element("Total", "Consolidated")
    for element_number in range(CELL_UPDATES):
        hierarchy.add_component(parent_name="Total", component_name=f"Element {str(element_number)}", weight=1)
    dimension = Dimension(DIMENSION_NAMES[0], hierarchies=[hierarchy])
    tm1.dimensions.update_or_create(dimension)

    hierarchy = Hierarchy(DIMENSION_NAMES[1], DIMENSION_NAMES[1])
    hierarchy.add_element("Element 1", "Numeric")
    dimension = Dimension(DIMENSION_NAMES[1], hierarchies=[hierarchy])
    tm1.dimensions.update_or_create(dimension)

    cube = Cube(CUBE_NAME, dimensions=DIMENSION_NAMES)
    tm1.cubes.update_or_create(cube)


def create_and_run(tm1):
    process = Process(
        name="TM1py Process",
        prolog_procedure="\r".join([
            f"CellPutN(1, '{CUBE_NAME}', 'Element {e}', 'Element 1');"
            for e
            in range(CELL_UPDATES)]))

    tm1.processes.update_or_create(process)

    success, status, error_log_file = tm1.processes.execute_with_return(process.name)

    if not success:
        raise RuntimeError("Failed to execute process")


def run_unbound(tm1):
    process = Process(
        name="",
        prolog_procedure="\r".join([
            f"CellPutN(1, '{CUBE_NAME}', 'Element {e}', 'Element 1');"
            for e
            in range(CELL_UPDATES)]))

    success, status, error_log_file = tm1.processes.execute_process_with_return(process)
    if not success:
        raise RuntimeError("Failed to execute unbound process")


with TM1Service(address="", port=12354, ssl=True, user="admin", password="apple") as tm1:
    prep(tm1)

    create_and_run(tm1)

    run_unbound(tm1)

[MariusWirtz]

resolved with #882