Opening new file with same name loads old file

[keflavich]

I've been seeing problems where I change a file on disk (delete it, re-run clean with entirely different parameters) then reload it, and somehow the old file - which does not exist any more - is shown in CARTA. Is there a caching system responsible for this?

[kswang1029]

please refer to CARTAvis/carta-backend#579
This is scheduled for the next release (v2.0)

[Jordatious]

I actually think this behaviour is preferable, for the comparison of a reprocessed file that is overwritten while the old one is still open.

[keflavich]

@Jordatious I don't think the current behavior is desirable. But I agree, I often do want the old one still open. The problem is, if I do:

• open file blah.fits (v1)
• edit file blah.fits so it is now v2
• open file blah.fits (v2) again

both images are identical in CARTA, and they are both the first version (v1).

What I'd like to see is that the first version stays open, but the newly-opened file is also shown - so there are two different image planes with the same name (or maybe a timestamp could be added to the name...). Is this also the behavior you're asking for?

[Jordatious]

@keflavich - oh I see! I haven't observed this behaviour. For me, step 3 opens (I presume you mean append) v2, while keeping v1 still loaded.

[keflavich]

Yes, I'm referring to Append only

[sanbw]

I see a similar issue.

I open an image using File->Open menu. The close it using File->Close menu. Delete that file and recreate it with different imaging parameters that does change the pixel values. When I load it again in CARTA using File->Open, the pixels displayed are from the old file which does not exist on the disk any more.

I then copied the current imaging on the disk to a new image on the disk. When I load this new image, I see the new pixel values.

This is CARTA running in Google Chrome browser talking to a CARTA server on a remote machine. The image is also on a remote machine.

Is there a workaround in CARTA for this?

The workaround mentioned earlier in this ticket suggests that File->Close be used before File->Open. However in my case, there is no image already loaded in CARTA. File->Close was called earlier and when re-loading the updated image, File->Close is not available. I am not appending the image. This is the only image being opened and closed.

[sanbw]

Could the problem that I am seeing be due to caching in the browser?

While I haven't had time to check this systematically, once I remove cached data from the browser (Chrome in my case), image with the same name when re-loaded after it was changed on the disk shows the changed pixels in CARTA as well.

And if this is the reason (browser caching), could CARTA code be fixed to remove the cached data in the browser related to the image name being loaded? It will be a massive help. The current behavior leads to a lot of user errors/misinterpretation.

[sanbw]

This problem (CARTA showing old pixels when re-load images with the same name, even though the image on the disk has changed) continues to persist with CARTA 2.0. I have tried using it from Firefox and Chrome. Also, I am using it in the "remote" mode. I.e. the CARTA server is running on a remote machine and I am viewing images across an internet connection.

[veggiesaurus]

This problem (CARTA showing old pixels when re-load images with the same name, even though the image on the disk has changed) continues to persist with CARTA 2.0. I have tried using it from Firefox and Chrome. Also, I am using it in the "remote" mode. I.e. the CARTA server is running on a remote machine and I am viewing images across an internet connection.

I'm trying to reproduce this issue on my development machine. The comments above mention FITS files, but the other issue linked specifies CASA images.

@sanbw any chance you have a simple example that I can follow step by step to reproduce the issue? (including details on the file types and how the files are written)

[veggiesaurus]

I have a set of casa images from @kswang1029 with the real, imaginary, amplitude and phase in separate image folders. I tried the following:

1. copy HD163296_CO_2_1_amplitude.image to qa/testing.image
2. run md5sum qa/testing.image/table.f0_TSM0 and note down the MD5
3. open qa/testing.image in CARTA, save an image of the histogram and the image view using clip bounds of [0, 1] (See figures below)
4. overwrite the existing data using dd:
   d if=HD163296_CO_2_1_phase.image/table.f0_TSM0 of=qa/testing.image/table.f0_TSM0
5. run md5sum qa/testing.image/table.f0_TSM0 to confirm that the MD5 has changed.
6. open qa/testing.image in CARTA again (File->Open File, without closing the old one first)
7. Save an image of the histogram and image view (using the same clip bounds as above).

initial images:

images after running dd and opening:

I'm not really sure how to reproduce the issue properly, because the method I used to overwrite the file shows that it behaves properly. I also tried just using cp, or even removing the old file first and then using cp. So I'm stumped

[veggiesaurus]

I also tried the same thing with FITS files and it still seemed to work just fine. Even appending the new file worked fine

[kswang1029]

The way to reproduce it reliably (at least on my Mac) is the following (using my demo data and script):

1. run clean_line.py as casapy -c clean_line.py. It will perform tclean and generate a set of images.
2. Once the script run to the end, launch CARTA and load the image molecular_line_clean.image and switch to channel 26 just for a better view of the source structure for comparison later. Keep CARTA open and do not close the image.
3. edit the script and use the alternative my_weight parameter in line 3 and 4 (now use "uniform" weighting). Then re-run the script casapy -c clean_line.py. The script will delete old image files first before generating new ones using the new my_weight parameter.
4. Once the script run to the end, use CARTA to "append" the same image molecular_line_clean.image and switch to channel 26. Now if we switch between the two images, we will see identical source features which should not be the case as we used a different weighting scheme to generate the cube.
5. Now, close the two image in CARTA and load molecular_line_clean.image again and switch to channel 26. We will see a different source feature.

demo script and dataset: https://drive.google.com/file/d/1zmrQBJJ_gjxb8p1rA6sBmSLSBOIEs7LV/view?usp=sharing
video record of all the above steps (download it first and watch for better resolution): https://drive.google.com/file/d/1qit_lynS5E1Fy2X9rd4ir7jkGhxJorsn/view?usp=sharing

I also tried to keep the first image animating and rerun the script. Surprisingly (or not), even the first image has been deleted by the script, the animation playback is not affected at all. So there must be some kind of caching mechanism (from casacore or OS perhaps) as the root cause of this issue.

[veggiesaurus]

To clarify, according to our tests, this only occurs when opening the same file twice without closing it. So under most circumstances, simply using File -> Open File should be fine, as it will close all open files first.

However, this doesn't solve the very common case of opening the file in a CARTA tab while another tab is open with the same file. In other words:

1. open CARTA tab, open file
2. overwrite file
3. open a second CARTA tab, open file again. The original file will not get closed because it's part of a separate CARTA session
4. Caching problem occurs

[sanbee]

The caching problem also shows up if one loads an image with the same name but a different directory. An often-used case for me. Also, when "appending" files, there is no automatic closure of files and this problem shows up when appending a file with the same name from a different directory.

In general, caching based on something as non-unique is the file name, seems to be ripe for bugs/confusing issues.

I suggest that the priority of fixing this be raised high. Presence of this makes CARTA very unreliable for precision work requiring comparing images.

[kswang1029]

@sanbee as a note here: casaviewer has the same behavior. CARTA uses similar codes to load images.

[sanbee]

@kswang1029: Not sure what you mean by "casaviewer has the same behavior". AFAICT, casaviewer does not show the caching issue being discussed here. It can load two different CASA images with the same name and it shows them both correctly. Also, it does not cache -- so once an image is closed/unloaded and reloaded, it shows the image as it is on the disk at the time of reloading.

Or perhaps I misunderstood your comment?

[kswang1029]

@kswang1029: Not sure what you mean by "casaviewer has the same behavior". AFAICT, casaviewer does not show the caching issue being discussed here. It can load two different CASA images with the same name and it shows them both correctly. Also, it does not cache -- so once an image is closed/unloaded and reloaded, it shows the image as it is on the disk at the time of reloading.

Or perhaps I misunderstood your comment?

please refer to the video https://drive.google.com/file/d/1K4bb74CGuI4OJ4cQr-6wNlf_OsOg_242/view?usp=sharing
using the same steps I mentioned above, with casaviewer we can see the exactly the same caching issue.

[sanbee]

It is nearly impossible to understand in detail what's being done in the video. However two points I'll make:

1. The issue I reported is for continuum images (single frequency plane). The video seems to be for a image cube. I haven't checked casaviewer for that usage and it may have the same issue, I just don't know. For continuum images, the problem is demonstrable.
2. Irrespective, this bug needs fixing in CARTA. CARTA is client-server model, and therefore it isn't re-started often. Caching (of any kind) is such an application needs to be robust. Caching in CARTA (whatever the source) does not seem to be robust and that makes it unreliable as an image viewer.

[kswang1029]

It is nearly impossible to understand in detail what's being done in the video. However two points I'll make:

1. The issue I reported is for continuum images (single frequency plane). The video seems to be for a image cube. I haven't checked casaviewer for that usage and it may have the same issue, I just don't know. For continuum images, the problem is demonstrable.
2. Irrespective, this bug needs fixing in CARTA. CARTA is client-server model, and therefore it isn't re-started often. Caching (of any kind) is such an application needs to be robust. Caching in CARTA (whatever the source) does not seem to be robust and that makes it unreliable as an image viewer.

@sanbee I am not saying we will not fix the problem. I just wanted to comment that this caching issue exists in the casaviewer too as far as I can see. CARTA used similar codes to handle casa images so CARTA has the issue too.
We have started some discussions internally these days about possible solutions as now we have a robust way to reproduce the problem. Solutions may be 1) apply fix in casacore codebase or 2) apply cache validation in carta codebase.

[sanbee]

@kswang1029 : I see your point. If caching is done in CASACore, which then shows up in any viewer that uses that code, fixing in CASACore may be the right approach (if practically possible). Thanks for the clarification.

I realize that I haven't used casaviewer in a "server" kind of a mode (if it is at all possible). I like CARTA's model a lot because it is client-server. This bug however makes me restart it every time I need to view the image (of the same name as the previous one), and takes away a lot of advantages (e.g. takes almost as long to see images in CARTA as it would to display casaviewer window across an internet connection). But of course, once a correct display is achieved with CARTA there are very significant advantages of using it over casaviewer.

[kswang1029]

@sanbee Before we fix the problem, could you try to close the image first before loading a new one with the same file name? Based on our test, this should bypass the issue. If you use File->Load to load the image, CARTA will issue a close_file command automatically so the issue should not exist. However, if you use File->Append to load the newly generated image with identical file name, the issue remains.

[veggiesaurus]

The caching problem also shows up if one loads an image with the same name but a different directory. An often-used case for me. Also, when "appending" files, there is no automatic closure of files and this problem shows up when appending a file with the same name from a different directory.

In general, caching based on something as non-unique is the file name, seems to be ripe for bugs/confusing issues.

I suggest that the priority of fixing this be raised high. Presence of this makes CARTA very unreliable for precision work requiring comparing images.

@sanbee I'm struggling to reproduce this particular issue on my machine. Could you attach some example images that are used to demonstrate this? Not being able to load files with the same file name is a more serious problem than correctly handling files that are overwritten while open in CARTA.

I tried the following:

• Create two CASA images that are identical in every way, other than the data. Images named testing.image but reside in two different folders (noise0 and noise1)
• Open CARTA in a new tab and load the noise0 image
• Open CARTA in another new tab and load the noise1 image without closing the other tab, or closing the file in the other tab

This works fine for me, so I'm not sure what I'm doing wrong 🤔

[sanbee]

@veggiesaurus , @kswang1029 : I'll try to get you some images and a possible route to reproduce the problem. But it looks like the earliest I can do is next week. Most of our computers here are down (UPS maintenance work in progress), and will come back online on Sunday at the earliest.

[kswang1029]

The caching problem also shows up if one loads an image with the same name but a different directory. An often-used case for me. Also, when "appending" files, there is no automatic closure of files and this problem shows up when appending a file with the same name from a different directory.
In general, caching based on something as non-unique is the file name, seems to be ripe for bugs/confusing issues.
I suggest that the priority of fixing this be raised high. Presence of this makes CARTA very unreliable for precision work requiring comparing images.

@sanbee I'm struggling to reproduce this particular issue on my machine. Could you attach some example images that are used to demonstrate this? Not being able to load files with the same file name is a more serious problem than correctly handling files that are overwritten while open in CARTA.

I tried the following:

• Create two CASA images that are identical in every way, other than the data. Images named testing.image but reside in two different folders (noise0 and noise1)
• Open CARTA in a new tab and load the noise0 image
• Open CARTA in another new tab and load the noise1 image without closing the other tab, or closing the file in the other tab

This works fine for me, so I'm not sure what I'm doing wrong 🤔

I tried a case that an imaging script generates a cube first, then make a new folder at the current directory where the cube resides, then the script generates another cube with identical filename in the new folder. After two cubes are generated, I load the two cubes in CARTA (so the file list widget shows two images with identical file name). I cannot reproduce the issue however 🤔