Top and bottom halves of data are inverted

[andrealorenzon]

Using bruker.read() to read from folders with Bruker experiments.

For some experiments, everything seems fine.

In other cases, data appear to be modified:

in this particular case, I expected the array to look like this:

so, with top and bottom half inverted.

I can hardcode a fix with a np.roll(data, -int(data.shape[0]/2), axis=0) but I wanted to understand why it happens only sometimes. Could there be something I forgot?

[kaustubhmote]

This again seems like an issue with the way quadrature in handled the indirect dimension while processing data using nmrglue. Can you share the scripts to help debug this?

[andrealorenzon]

sure.

This is my pipeline function:

def runPipeline (dic: dict, data: ndarray, dmxFlag: bool, 
                PS1p0:float, PS1p1:float, 
                PS2p0:float, PS2p1:float):
      
    import nmrglue as ng

    try:
        os.remove('.data/tempSave/outputFile.ft2')
        print("removed old outputFile")
    except:
        print("no file to remove")
        

    dic, data = ng.pipe_proc.sp(dic, data,off=0.5, end=0.95, pow=2, c=0.5)
    dic, data = ng.pipe_proc.zf(dic, data, zf=1, auto=True)
    dic, data = ng.pipe_proc.ft(dic, data)

    N=dic.get("FDDMXVAL",0)

    if not dmxFlag:
        dic, data = ng.pipe_proc.ps(dic, data, p0=0, p1=-360*N,)

    dic, data = ng.pipe_proc.ps(dic, data, p0=PS1p0, p1=PS1p1)
    dic, data = ng.pipe_proc.di(dic, data)
    dic, data = ng.pipe_proc.tp(dic, data) 
    dic, data = ng.pipe_proc.sp(dic, data, off=0.5, end=0.95, pow=2, c=1)
    dic, data = ng.pipe_proc.zf(dic, data, auto=True,)
    dic, data = ng.pipe_proc.ft(dic, data, alt=True)
    dic, data = ng.pipe_proc.ps(dic, data, p0=PS2p0, p1=PS2p1,)
    dic, data = ng.pipe_proc.di(dic, data)
    dic, data = ng.pipe_proc.tp(dic, data)

    return dic, data

I think that I missed something about how to get from dic, data, or from the original data source, all the information needed.

The data themselves are retrieved with this script, from the uploaded folder (it runs in a Streamlit app):

def openZipFromTmpPath(source: str):
    """retrieves dic, data from folder"""
    import fs
    import os
    import nmrglue as ng
    from zipfile import ZipFile
    from pathlib import Path
    from fs.tempfs import TempFS
    from fs.zipfs import ZipFS
    import streamlit as st
    
    tmp_fs = fs.open_fs("temp://fidViewerBuffer")

    with tmp_fs:
        with fs.zipfs.ZipFS(source) as zip_fs:
            fs.copy.copy_fs(zip_fs,tmp_fs)
        all_dirs = os.listdir('/tmp/')
        res = filter(lambda x: 'fidViewer' in x, all_dirs)
        for r in res:
            p = os.path.join('/','tmp',r)
            folder = list(os.walk(p))[2][0]
            tempDic, tempData = ng.bruker.read(folder)
            return tempDic, tempData


def readZIP(uploaded_file):
    """ unzip in temp directory"""
    import fs
    import os
    import nmrglue as ng
    from zipfile import ZipFile
    from fs.tempfs import TempFS
    from fs.zipfs import ZipFS
    import streamlit as st
    
    try:
        dic, data = openZipFromTmpPath(uploaded_file)
        data = ng.fileio.bruker.remove_digital_filter(dic, data)
        #convert from bruker to pipe
        udic = ng.bruker.guess_udic(dic, data)
        C = ng.convert.converter()
        C.from_bruker(dic, data, udic)
        dic, data = C.to_pipe()
        return dic, data
    
    except Exception as e:
        st.error(e)

The idea is to be able to open correctly ANY kind of 2D spectra, dynamically retrieving all required parameters from the files themselves. If any additional parameter has to be specified by the user, I'll make some control to enter its value. Problem is, depending from my lack of topic knowledge, for sure, to be able to understand which parameters can be inferred by nmrglue from the files, and which are to be inserted by the user, and where, to obtain a correctly oriented/rolled/processed data, and there's where I need some help.

[kaustubhmote]

The processing seems fine to me. Is the data from a sensitivity enhanced experiment? If so, you may need to do reshuffling of the data (Echo-Antiecho or Rance-Kay) before you process it. Normally, in nmrpipe, this is done at the conversion step, so you should look at your fid.com file and see if it says either of those two things. If this is the case, nmrglue has a function in nmrglue/process/nmrtxt/rance_kay.py which you can use. This parameter should also be in the dic["acqus"]["FnMODE"].

[andrealorenzon]

These are 2 examples of fid.com files.

For data correctly visualized:

#!/bin/csh

bruk2pipe -verb -in ./ser \
  -bad 0.0 -ext -aswap -AMX -decim 1680 -dspfvs 21 -grpdly 76  \
  -xN              4096  -yN              1024  \
  -xT              2048  -yT               512  \
  -xMODE            DQD  -yMODE        Complex  \
  -xSW        11904.762  -ySW        12003.656  \
  -xOBS         600.183  -yOBS         600.183  \
  -xCAR           4.771  -yCAR           4.771  \
  -xLAB             1Hx  -yLAB             1Hy  \
  -ndim               2  -aq2D          States  \
| nmrPipe -fn MULT -c 3.12500e+01 \
  -out ./test.fid -ov

and for data that are not correctly visualized:

#!/bin/csh

bruk2pipe -verb -in ./ser \
  -bad 0.0 -ext -aswap -AMX -decim 1680 -dspfvs 20 -grpdly 68  \
  -xN              2048  -yN               800  \
  -xT              1024  -yT               400  \
  -xMODE            DQD  -yMODE  Echo-AntiEcho  \
  -xSW        11904.762  -ySW        15093.772  \
  -xOBS         600.183  -yOBS         150.938  \
  -xCAR           4.773  -yCAR         150.715  \
  -xLAB              1H  -yLAB             13C  \
  -ndim               2  -aq2D         Complex  \
| nmrPipe -fn MULT -c 8.33333e+00 \
  -out ./test.fid -ov

I see the Echo-AntiEcho in the latter.

This parameter should also be in the dic["acqus"]["FnMODE"].

At the moment that parameters is == 0 for both source files. It's still unclear how can I infer the correct procedure from dic content.

[kaustubhmote]

Oh, I should have said dic["acqu2s"]["FnMODE"] since this is the indirect dimension. Echo-AntiEcho should be == 6. The numbers correspond to the order in which the options appear in Topspin's drop-down menu for this parameter in the ACQUS tab.

[andrealorenzon]

dic["acqu2s"]["FnMODE"] == 5 for NOESY data, and == 6 for HSQC data.. great so know I know what can I infer the mode from. :) The only info I still miss is which transformation should I apply in either case to fix the reading of data.

[kaustubhmote]

You can look at the file nmrglue/process/nmrtxt/rance_kay.py for the function that reshuffles the data. This seems to be unavailable directly from any module. Perhaps the best solution is to reshuffle the data manually right after it is read in.

[andrealorenzon]

Ok, I'll look into it. Thanks.

EDIT: great, it fixed the issue. Thank you.