Refactored bulk uploading of objects

[ajstewart]

• Moved to generator and iteration uploading.
• Removed the storage of meas_dj_obj and src_dj_obj.
• Links made on id values instead of objects.

Fixes #381.

[ajstewart]

@srggrs @marxide After a successful full test this is ready. You'll see that I put the generator functions into their own new file. However I am also happy to move these to somewhere like loading.py if you think that's better.

[ajstewart]

Just to record some of the information in this PR, below is a before and after plot of RAM usage on a full tiles run of the vast pilot survey (the after run does not have the image injection stage):

The high two epoch creation and upload highlighted the memory issue that led to these changes.

[srggrs]

I think we should aim to do 2 things:
1 - pass the dataframe to upload* and inside that create the id column
2 - aim at building a generator that would return the chunk directly, see example above

def souce_model_f(row, pipe_run_id):
    name = (
        f"ASKAP_{deg2hms(row['wavg_ra'])}"
        f"{deg2dms(row['wavg_dec'])}".replace(":", "")
    )
    src = Source()
    src.run_id = pipe_run_id
    src.name = name
    for fld in src._meta.get_fields():
        if getattr(fld, 'attname', None) and fld.attname in row.index:
            setattr(src, fld.attname, row[fld.attname])

    return src

def source_model_generator(chuck: pd.DataFrame):
    models = chuck.apply(lambda row: souce_model_f(row), axis=1)
    return models.tolist()


## MAIN LOGIC
for idx in range(0, size, batch_size):
    models = source_model_generator(src_df.iloc[idx: idx + batch_size])
    ids = upload_source(models, return_ids=True)

Potentially we can use yield in yield models.tolist() and make generators, then call them using the list(mygenerator)

Open to discussion @ajstewart @marxide

[ajstewart]

I think we should aim to do 2 things:

1 - pass the dataframe to upload* and inside that create the id column

2 - aim at building a generator that would return the chunk directly, see example above

def souce_model_f(row, pipe_run_id):

    name = (

        f"ASKAP_{deg2hms(row['wavg_ra'])}"

        f"{deg2dms(row['wavg_dec'])}".replace(":", "")

    )

    src = Source()

    src.run_id = pipe_run_id

    src.name = name

    for fld in src._meta.get_fields():

        if getattr(fld, 'attname', None) and fld.attname in row.index:

            setattr(src, fld.attname, row[fld.attname])



    return src



def source_model_generator(chuck: pd.DataFrame):

    models = chuck.apply(lambda row: souce_model_f(row), axis=1)

    return models.tolist()





## MAIN LOGIC

for idx in range(0, size, batch_size):

    models = source_model_generator(src_df.iloc[idx: idx + batch_size])

    ids = upload_source(models, return_ids=True)

Potentially we can use yield in yield models.tolist() and make generators, then call them using the list(mygenerator)

Open to discussion @ajstewart @marxide

So compared to how I've done it now, you'd like just the data frame passed to upload function which then does the generator and upload? Though it looks like you don't think 'yield' is needed?

[ajstewart]

@srggrs I've moved things around a bit to make more sense and be consistent with the measurements and images upload. All the generators are now loaded in loading.py only (apart from forced fits upload) and only the data frames are passed to the upload functions (apart from sources which also needs the Run).

I tried to use a for loop with islice but for some reason I couldn't get it to go which confused me.

[srggrs]

I think we can leave the generators in that file, maybe call it model_generator.py then move the id column creation inside the upload function and rename the functions make_upload_

[ajstewart]

@srggrs I think I've addressed all the comments above in a2c8224 apart from the loop (see my comment on that).

p.s. I didn't have time to do any docstrings at the moment sorry.

EDIT. Forgot to add that the latest commits also contain a fix to newsources.py where a reference to numpy has snuck in where it should np.