can't compile casatables_impl on arm64: narrowing conversion from 'int' to 'char' #345

d3v-null · 2024-03-09T08:02:51Z

Hey Peter,
I've been trying to get rubbl compiling within an arm64 docker image. No need to test this on arm hardware, the latest version of Docker desktop will automatically emulate it for you.

Here's a dockerfile:

ARG UBUNTU_VERSION=20.04

FROM ubuntu:${UBUNTU_VERSION} as base

ENV DEBIAN_FRONTEND="noninteractive"
RUN apt-get update && \
    apt-get install -y \
        aoflagger-dev \
        build-essential \
        clang \
        curl \
        git \
        jq \
        lcov \
        libcfitsio-dev \
        liberfa-dev \
        libssl-dev \
        pkg-config \
        unzip \
        wget \
        zip \
    && \
    apt-get clean all && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
    apt-get -y autoremove

# Get Rust
ARG RUST_VERSION=stable
ENV RUSTUP_HOME=/opt/rust CARGO_HOME=/opt/cargo PATH=/opt/cargo/bin:$PATH
RUN mkdir -m755 $RUSTUP_HOME $CARGO_HOME && \
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y \
        --profile=minimal \
        --default-toolchain=${RUST_VERSION}-$(uname -m)-unknown-linux-gnu

# this is just for some test data
RUN git clone --depth 1 https://github.com/MWATelescope/Marlu.git /marlu

ARG RUBBL_VERSION=0.3.0
RUN git clone --depth 1 --branch rubbl@${RUBBL_VERSION} https://github.com/pkgw/rubbl.git /rubbl && \
    cd /rubbl && \
    cargo run --example tableinfo -- /marlu/tests/data/1254670392_avg/1254670392.cotter.none.trunc.ms

if you haven't already, you'll need to set up docker buildx with

docker buildx create --driver=docker-container --name=multi --use

Building this on amd64 works fine: (but the example itself has a bug)

docker buildx build --platform linux/amd64 .
...
warning: `rubbl_casatables` (lib) generated 1 warning
    Finished dev [unoptimized + debuginfo] target(s) in 57.73s
     Running `target/debug/examples/tableinfo /marlu/tests/data/1254670392_avg/1254670392.cotter.none.trunc.ms`
thread 'main' panicked at casatables/examples/tableinfo.rs:27:34:
Mismatch between definition and access of `IN-TABLE`. Could not downcast to std::path::PathBuf, need to downcast to alloc::string::String

but it fails to compile on arm64:

docker buildx build --platform linux/arm64 .

error occurred: Command "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Dcasacore=rubbl_casacore" "-DUSE_THREADS=1" "-o" "/rubbl/target/debug/build/rubbl_casatables_impl-5d357e254cf076d0/out/casacore/casa/OS/IBMConversion.o" "-c" "casacore/casa/OS/IBMConversion.cc" with args "c++" did not execute successfully (status code exit status: 1).

cargo:warning=casacore/casa/OS/IBMConversion.cc:72:5: error: narrowing conversion of '-26' from 'int' to 'char' [-Wnarrowing]
cargo:warning=   72 |     };
cargo:warning=      |     ^
cargo:warning=casacore/casa/OS/IBMConversion.cc:72:5: error: narrowing conversion of '-25' from 'int' to 'char' [-Wnarrowing]
...
cargo:warning=casacore/casa/OS/IBMConversion.cc:72:5: error: narrowing conversion of '-95' from 'int' to 'char' [-Wnarrowing]
cargo:warning=casacore/casa/OS/IBMConversion.cc: In static member function 'static void rubbl_casacore::IBMConversion::fromLocal(void*, char)':
cargo:warning=casacore/casa/OS/IBMConversion.cc:95:14: warning: comparison is always false due to limited range of data type [-Wtype-limits]
cargo:warning=   95 |     if (from < 0) {
cargo:warning=      |         ~~~~~^~~

In upstream casacore, the type of toEBCDIC has changed from static char to static signed char https://github.com/casacore/casacore/blob/c6c5513af42b0fd2aaa8d342e876ba1b05542b0a/casa/OS/IBMConversion.cc#L53

so it might just be a matter of updating the casacore source.

Cheers!

The text was updated successfully, but these errors were encountered:

d3v-null · 2024-03-09T10:04:01Z

unfortunately when I try running update.sh I get a few missing files, so things have obviously moved around a little bit between 3.1.1 and 3.5.0 casacore/casacore@v3.1.1...v3.5.0

cp: cannot stat '/home/dev/src/casacore/casacore/casa/Arrays/ArrayIO.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Arrays/ArrayIO.tcc': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Arrays/IPosition2.cc': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Arrays/LogiArrayFwd.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/BasicSL/RegexBase.cc': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/BasicSL/RegexBase.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/OS/Mutex.cc': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Utilities/RegSequence.cc': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Utilities/RegSequence.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Utilities/Register.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Utilities/Register.tcc': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Utilities/cregex.cc': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Utilities/cregex.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/version.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/tables/Tables/ArrColData.tcc': No such file or directory

Some notes from looking through the changes:

casa/Arrays/ArrayIO → casa/IO/ArrayIO
casa/Arrays/IPosition2 → casa/IO/IPositionIO
casa/Arrays/LogiArrayFwd → casa/Arrays/ArrayFwd
casa/BasicSL/RegexBase, casa/Utilities/cregex → removed for std::regex
casa/OS/Mutex → removed for std::mutex (concurrency might actually work now lol)
casa/Utilities/RegSequence → ??
casa/Utilities/Register.tcc → tables/DataMan/StManColumnBase.cc ??
casa/version.h → casa/version.h.in ??
tables/Tables/ArrColData.tcc → tables/Tables/ArrColData.cc
new:

casa/Arrays/Storage
casa/Arrays/ArrayStr
casa/Arrays/Memory

pkgw · 2024-03-12T00:32:21Z

We are probably due for an update of the casacore source. Do you think you might be able to wrangle the updates to update.sh or would you like me to take a look at it?

d3v-null · 2024-03-12T07:29:12Z

Thanks Peter,
I spent a couple of hours looking at it, and didn't get very far. It's a bit tricky to test it as well, since from my understanding, I need to rebase onto vendor-casacore each time I want to test changes to the casacore source, and since some of the rebased commits involve changes to casacore source, I'm not entirely sure I rebased everything correctly. If you wouldn't mind having a look, that would be great.

pkgw · 2024-03-12T13:11:07Z

OK, I'll try to take a look at it, and also think about how to make things more maintainable. Feel free to ping me about this since it's the kind of thing that might slip off my todo list.

d3v-null · 2024-04-16T07:10:45Z

Hey Peter,
I wanted to have another crack at this, but failed again, so I thought I'd show you what I did and what issues I encountered and ask if you have any advice.

This time I tried one minor release at a time, starting with upgrading from v3.1.1 to v3.2.1 (at least, I think vendor-casacore is v3.1.1 based on casatables_impl: update to casacore 3.1.1 )

The plan:

checkout vendor-casacore
update with src from v3.2.1
rebase main onto vendor-casacore

This should be simple, because there are only 2 commits that affect files under casatables_impl since the head of vendor-casacore:

but for some reason, I get many many conflicts when I do my rebase.

You don't need to read all this

Because I'm doing this on ubuntu22.04, I can't cmake casacore 3.2.0, so I did it in a docker image:

# casacore-ubuntu18-gcc.Dockerfile
FROM ubuntu:18.04
RUN export DEBIAN_FRONTEND=noninteractive \
    && apt-get update \
    && apt-get install -y \
    gcc \
    g++ \
    cmake \
    flex \
    bison \
    libblas-dev \
    liblapack-dev \
    libcfitsio-dev \
    wcslib-dev \
    libfftw3-dev \
    gfortran \
    libncurses5-dev \
    libreadline6-dev \
    libhdf5-serial-dev \
    libboost-dev \
    libboost-python-dev \
    libboost-test-dev \
    libgsl-dev \
    python-numpy \
    python3-numpy \
    wget \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
# Install WSRT Measures (extra casacore data, for tests)
# Note: The file on the ftp site is updated daily. When warnings regarding leap
# seconds appear, ignore them or regenerate the docker image.
    && wget -nv -O /WSRT_Measures.ztar ftp://ftp.astron.nl/outgoing/Measures/WSRT_Measures.ztar \
    && mkdir -p /usr/local/share/casacore/data \
    && cd /usr/local/share/casacore/data \
    && tar xfz /WSRT_Measures.ztar \
    && rm /WSRT_Measures.ztar

cd ~/src/casacore
mkdir build
git checkout v3.2.1
docker build -f casacore-ubuntu18-gcc.Dockerfile -t d3vnull0/casacore-builder-ubuntu18-gcc .
docker run -it -v ${PWD}:${PWD} -w ${PWD} d3vnull0/casacore-builder-ubuntu18-gcc /bin/bash -c "cd build; cmake .."
cd ~/src/rubbl
git checkout vendor-casacore
casatables_impl/update.sh ~/src/casacore/build
# cp: cannot stat '/home/dev/src/casacore/casacore/casa/version.h': No such file or directory
rm casatables_impl/casacore/casa/version.h
git stage casatables_impl
git commit -m 'casatables_impl: update to casacore 3.2.1'
git checkout master
git rebase vendor-casacore

When I do this, I get a conflict with this commit: casatables_impl/casacore/casa/System/Aipsrc.cc: attempt to defuse config-file reading

I resolve this by making both Aipsrc::find and Aipsrc::findNoParse return False.

Then I continue the rebase, but each commit runs into a new conflict:
casatables_impl: fix warnings encountered with GCC 7.2 - Aipsrc::findNoParse is further modified, I accept the current change each time, and delete files that have been deleted, and progress through the rebase.

casatables_impl: fix warnings encountered with GCC 7.2 - I accept the incoming changes

then I keep going for a few more conflicts until I get to this one, and I just don't understand how git thinks this is a conflict , and how to deal with it.

d3v-null · 2024-05-16T13:27:50Z

I've made another attempt.
This time, I'm exploring what changes are needed for each casacore version, to figure out how I can attempt the rebase again.

The plan:

checkout master
update casacore source to 3.2.1
see what changes are needed to get the tests passing
repeat for 3.3.0, 3.4.0, 3.5.0

tl;dr

For the update to casacore 3.2.1 and 3.3.0, simply reverting the files back to their state in master worked, since they have not received any additional changes. These are ready to go, although I can't see a lot in these changes that would be particularly valuable, except maybe ADIOS2 support.

3.4.0 adds a lot of important modernizations, but it's also where things get messy. I've written some notes in the hope that this might help. The problem I'm stuck on are these cyclic dependencies. Basically IPosition.h and Slice.h reference Vector.h, but don't include it, and Vector.h includes Array.h, which includes IPosition.h. I don't know how this mess even compiles in the first place.

For Arm support we need 3.5.0

Lots of boring details

changes between vendor-casacore and master

casacore-3.2.1

casacore diff
- Remove some unused functionality
- Prevent setting OpenMP num_threads to zero
- Fix locking issue with IncrementalStorageManager
- Various improvements to TaQL
- Various ADIOS2 improvements

get casacore source

export CASACORE_VER=3.2.1
cd ~/src/casacore
sudo rm -rf build CMakeFiles CMakeCache.txt
git checkout v${CASACORE_VER}
docker build -f casacore-ubuntu18-gcc.Dockerfile -t d3vnull0/casacore-builder-ubuntu18-gcc .
mkdir build
docker run -it -v ${PWD}:${PWD} -w ${PWD}/build d3vnull0/casacore-builder-ubuntu18-gcc /bin/bash -c "cmake .."
cd ~/src/rubbl
git checkout master # sync'd my fork
git checkout -b casacore-${CASACORE_VER}
# patch update.sh to not care about the branch
sed -i 's/\[ "\$cur_branch" != vendor-casacore \]/false/g' casatables_impl/update.sh
casatables_impl/update.sh ../casacore/build

changes needed to get the tests passing:

# static_cast<void *> https://github.com/pkgw/rubbl/commit/bed1cea67ccc96564c0a1467c8b7d5a7f0644a5f
git checkout master -- casatables_impl/casacore/casa/Arrays/Array2Math.cc
# (char) -> static_cast<char>
git checkout master -- casatables_impl/casacore/casa/System/ObjectID2.cc
# buffer sizes https://github.com/pkgw/rubbl/commit/1bf45d29873accbde16db75cb5c8f6a66b9a9577
git checkout master -- casatables_impl/casacore/tables/Tables/ReadAsciiTable.cc
git checkout master -- casatables_impl/casacore/casa/Containers/RecordDescRep.cc
# impl =
git checkout master -- casatables_impl/casacore/casa/Quanta/MV*.h
# rubbl customization: never actually use any config files
git checkout master -- casatables_impl/casacore/casa/System/Aipsrc.cc
# remove TAQL references https://github.com/pkgw/rubbl/commit/79fe0b245d0537b4fdf837ef6cebd13be8661abe
git checkout master -- casatables_impl/casacore/tables/DataMan.h
git checkout master -- casatables_impl/casacore/tables/DataMan/{MSMIndColumn,DataManager}.cc
git checkout master -- casatables_impl/casacore/tables/Tables/{BaseTable,Table}.cc
git checkout master -- casatables_impl/casacore/tables/Tables/Table.cc
git checkout master -- casatables_impl/casacore/tables/Tables.h
# add fallthrough warnings
git checkout master -- casatables_impl/casacore/casa/{IO/RegularFileIO.cc,Utilities/cregex.cc,Utilities/Regex.cc,Utilities/ValType.cc}

tests pass! here's the diff between master and my 3.2.1 changes: d3v-null@a08dd11

casacore-3.3.0

casacore diff
- Replace implementation of complex trig functions by std::
- Fix bug SciMath / StatsFramework where parameters were passed incorrectly
- Pass sorting algorithm optio nthrough to table iterator
- Support for ADIOS2
- Add .casarc option to disable table locking

export CASACORE_VER=3.3.0
cd ~/src/casacore
sudo rm -rf build CMakeFiles CMakeCache.txt
git checkout v${CASACORE_VER}
mkdir build
docker run -it -v ${PWD}:${PWD} -w ${PWD}/build d3vnull0/casacore-builder-ubuntu18-gcc /bin/bash -c "cmake .."
cd ~/src/rubbl
git checkout -b casacore-${CASACORE_VER}
casatables_impl/update.sh ../casacore/build

# static_cast<void *> https://github.com/pkgw/rubbl/commit/bed1cea67ccc96564c0a1467c8b7d5a7f0644a5f
git checkout master -- casatables_impl/casacore/casa/Arrays/Array2Math.cc
# (char) -> static_cast<char>
git checkout master -- casatables_impl/casacore/casa/System/ObjectID2.cc
# buffer sizes https://github.com/pkgw/rubbl/commit/1bf45d29873accbde16db75cb5c8f6a66b9a9577
git checkout master -- casatables_impl/casacore/tables/Tables/ReadAsciiTable.cc
git checkout master -- casatables_impl/casacore/casa/Containers/RecordDescRep.cc
# impl =
git checkout master -- casatables_impl/casacore/casa/Quanta/MV*.h
# rubbl customization: never actually use any config files
git checkout master -- casatables_impl/casacore/casa/System/Aipsrc.cc
# remove TAQL references https://github.com/pkgw/rubbl/commit/79fe0b245d0537b4fdf837ef6cebd13be8661abe
git checkout master -- casatables_impl/casacore/tables/DataMan.h
git checkout master -- casatables_impl/casacore/tables/DataMan/{MSMIndColumn,DataManager}.cc
git checkout master -- casatables_impl/casacore/tables/Tables/{BaseTable,Table}.cc
git checkout master -- casatables_impl/casacore/tables/Tables/Table.cc
git checkout master -- casatables_impl/casacore/tables/Tables.h

tests pass again! diff from 3.2.1 to 3.3.0: d3v-null@0e130fd

casacore-3.4.0

casacore diff
- Major modernization of Arrays, bringing it more in line with C++11
- Use std::regex
- Remove many warnings
- Use FFTW as default FFT provider
- UVFITS improvements
- Support for 64-bit row numbers.
a note aboute CTDS changes
casa/CMakeLists.txt
tables/CMakeLists.txt

export CASACORE_VER=3.4.0
cd ~/src/casacore
sudo rm -rf build CMakeFiles CMakeCache.txt
git checkout v${CASACORE_VER}
mkdir build
docker run -it -v ${PWD}:${PWD} -w ${PWD}/build d3vnull0/casacore-builder-ubuntu18-gcc /bin/bash -c "cmake .."
cd ~/src/rubbl
git checkout -b casacore-${CASACORE_VER}
casatables_impl/update.sh ../casacore/build

now we get more missing files:

cp: cannot stat '/home/dev/src/casacore/casacore/casa/Arrays/ArrayIO.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Arrays/ArrayIO.tcc': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Arrays/IPosition2.cc': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Arrays/LogiArrayFwd.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/BasicSL/RegexBase.cc': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/BasicSL/RegexBase.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Utilities/cregex.cc': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/Utilities/cregex.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/casa/version.h': No such file or directory
cp: cannot stat '/home/dev/src/casacore/casacore/tables/Tables/ArrColData.tcc': No such file or directory

so we move the files to the new locations, stage them in git (so git ls-files sees them), then re-run update.sh

# move casa/Arrays/ArrayIO → casa/IO/ArrayIO
mv casatables_impl/casacore/casa/Arrays/ArrayIO.{h,tcc} casatables_impl/casacore/casa/IO/
sed -i 's/Arrays\/ArrayIO/IO\/ArrayIO/' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/IO/ArrayIO.{h,tcc}
# move casa/Arrays/IPosition2 → casa/IO/IPositionIO
mv casatables_impl/casacore/casa/Arrays/IPosition2.cc casatables_impl/casacore/casa/IO/IPositionIO.cc
sed -i 's/Arrays\/IPosition2/IO\/IPositionIO/' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/IO/IPositionIO.cc
# move casa/Arrays/LogiArrayFwd → casa/Arrays/ArrayFwd
mv casatables_impl/casacore/casa/Arrays/LogiArrayFwd.h casatables_impl/casacore/casa/Arrays/ArrayFwd.h
sed -i 's/Arrays\/LogiArrayFwd/Arrays\/ArrayFwd/' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/Arrays/ArrayFwd.h
# remove casa/BasicSL/RegexBase, casa/Utilities/cregex
rm casatables_impl/casacore/casa/BasicSL/RegexBase.{cc,h}
git stage casatables_impl/casacore/casa/BasicSL/RegexBase.{cc,h}
sed -i '/RegexBase/d' casatables_impl/build.rs
rm casatables_impl/casacore/casa/Utilities/cregex.{cc,h}
git stage casatables_impl/casacore/casa/Utilities/cregex.{cc,h}
sed -i '/cregex/d' casatables_impl/build.rs
# move tables/Tables/ArrColData.tcc → tables/Tables/ArrColData.cc
mv casatables_impl/casacore/tables/Tables/ArrColData.tcc casatables_impl/casacore/tables/Tables/ArrColData.cc
sed -i 's/ArrColData.tcc/ArrColData.cc/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/ArrColData.cc
# add casa/Arrays/Storage
touch casatables_impl/casacore/casa/Arrays/Storage.h
sed -i 's/casacore\/casa\/Arrays\/Slicer.h/casacore\/casa\/Arrays\/Slicer.h",\n    "casacore\/casa\/Arrays\/Storage.h/' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/Arrays/Storage.h
# add casa/Arrays/ArrayStr
touch casatables_impl/casacore/casa/Arrays/ArrayStr.h
sed -i 's/casacore\/casa\/Arrays\/ArrayPosIter.h/casacore\/casa\/Arrays\/ArrayPosIter.h",\n    "casacore\/casa\/Arrays\/ArrayStr.h/' casatables_impl/build.rs
touch casatables_impl/casacore/casa/Arrays/ArrayStr.tcc
sed -i 's/casacore\/casa\/Arrays\/ArrayPosIter.h/casacore\/casa\/Arrays\/ArrayPosIter.h",\n    "casacore\/casa\/Arrays\/ArrayStr.tcc/' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/Arrays/ArrayStr.{h,tcc}
# add casa/Arrays/Memory
touch casatables_impl/casacore/casa/Arrays/Memory.h
sed -i 's/casacore\/casa\/Arrays\/Matrix.tcc/casacore\/casa\/Arrays\/Matrix.tcc",\n    "casacore\/casa\/Arrays\/Memory.h/' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/Arrays/Memory.h
# add casa/Arrays/ElementFunctions.h
touch casatables_impl/casacore/casa/Arrays/ElementFunctions.h
sed -i 's/casacore\/casa\/Arrays\/Cube.tcc/casacore\/casa\/Arrays\/Cube.tcc",\n    "casacore\/casa\/Arrays\/ElementFunctions.h/' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/Arrays/ElementFunctions.h

# add tables/Tables/ArrayColumnBase.cc
touch casatables_impl/casacore/tables/Tables/ArrayColumnBase.cc
sed -i 's/casacore\/tables\/Tables\/ArrColDesc_tmpl.cc/casacore\/tables\/Tables\/ArrColDesc_tmpl.cc",\n    "casacore\/tables\/Tables\/ArrayColumnBase.cc/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/ArrayColumnBase.cc
# add tables/Tables/ArrColData.cc
touch casatables_impl/casacore/tables/Tables/ArrColData.cc
sed -i 's/casacore\/tables\/Tables\/ArrayColumnBase.cc/casacore\/tables\/Tables\/ArrayColumnBase.cc",\n    "casacore\/tables\/Tables\/ArrColData.cc/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/ArrColData.cc
# add tables/Tables/ArrColDesc.cc
touch casatables_impl/casacore/tables/Tables/ArrColDesc.cc
sed -i 's/casacore\/tables\/Tables\/ArrayColumnBase.cc/casacore\/tables\/Tables\/ArrayColumnBase.cc",\n    "casacore\/tables\/Tables\/ArrColDesc.cc/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/ArrColDesc.cc
# add tables/Tables/RowNumbers.cc
touch casatables_impl/casacore/tables/Tables/RowNumbers.cc
sed -i 's/casacore\/tables\/Tables\/RowCopier.cc/casacore\/tables\/Tables\/RowCopier.cc",\n    "casacore\/tables\/Tables\/RowNumbers.cc/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/RowNumbers.cc
# add tables/DataMan/DataManagerColumn.cc
touch casatables_impl/casacore/tables/DataMan/DataManagerColumn.cc
sed -i 's/casacore\/tables\/DataMan\/DataManager.cc/casacore\/tables\/DataMan\/DataManager.cc",\n    "casacore\/tables\/DataMan\/DataManagerColumn.cc/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/DataManagerColumn.cc
# add tables/DataMan/StManColumnBase.cc
touch casatables_impl/casacore/tables/DataMan/StManColumnBase.cc
sed -i 's/casacore\/tables\/DataMan\/StManColumn.cc/casacore\/tables\/DataMan\/StManColumn.cc",\n    "casacore\/tables\/DataMan\/StManColumnBase.cc/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/StManColumnBase.cc
# add tables/DataMan/VirtArrCol.cc
touch casatables_impl/casacore/tables/DataMan/VirtArrCol.cc
sed -i 's/casacore\/tables\/DataMan\/VirtColEng.cc/casacore\/tables\/DataMan\/VirtColEng.cc",\n    "casacore\/tables\/DataMan\/VirtArrCol.cc/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/VirtArrCol.cc
# add tables/DataMan/VirtScaCol.cc
touch casatables_impl/casacore/tables/DataMan/VirtScaCol.cc
sed -i 's/casacore\/tables\/DataMan\/VirtColEng.cc/casacore\/tables\/DataMan\/VirtColEng.cc",\n    "casacore\/tables\/DataMan\/VirtScaCol.cc/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/VirtScaCol.cc
# delete tables/Tables/ArrColData.tcc
git rm casatables_impl/casacore/tables/Tables/ArrColData.tcc
# add tables/Tables/ArrayColumnBase.h
touch casatables_impl/casacore/tables/Tables/ArrayColumnBase.h
sed -i 's/casacore\/tables\/Tables\/ArrayColumn.h/casacore\/tables\/Tables\/ArrayColumn.h",\n    "casacore\/tables\/Tables\/ArrayColumnBase.h/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/ArrayColumnBase.h
# add tables/Tables/RowNumbers.h
touch casatables_impl/casacore/tables/Tables/RowNumbers.h
sed -i 's/casacore\/tables\/Tables\/RowCopier.h/casacore\/tables\/Tables\/RowCopier.h",\n    "casacore\/tables\/Tables\/RowNumbers.h/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/RowNumbers.h
# add tables/DataMan/DataManagerColumn.h
touch casatables_impl/casacore/tables/DataMan/DataManagerColumn.h
sed -i 's/casacore\/tables\/DataMan\/DataManager.h/casacore\/tables\/DataMan\/DataManager.h",\n    "casacore\/tables\/DataMan\/DataManagerColumn.h/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/DataManagerColumn.h
# add tables/DataMan/StManColumnBase.h
touch casatables_impl/casacore/tables/DataMan/StManColumnBase.h
sed -i 's/casacore\/tables\/DataMan\/StManColumn.h/casacore\/tables\/DataMan\/StManColumn.h",\n    "casacore\/tables\/DataMan\/StManColumnBase.h/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/StManColumnBase.h
# add tables/DataMan/VACEngine.h
touch casatables_impl/casacore/tables/DataMan/VACEngine.h
sed -i 's/casacore\/tables\/DataMan\/VSCEngine.tcc/casacore\/tables\/DataMan\/VSCEngine.tcc",\n    "casacore\/tables\/DataMan\/VACEngine.h/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/VACEngine.h
# add tables/DataMan/VACEngine.tcc
touch casatables_impl/casacore/tables/DataMan/VACEngine.tcc
sed -i 's/casacore\/tables\/DataMan\/VSCEngine.tcc/casacore\/tables\/DataMan\/VSCEngine.tcc",\n    "casacore\/tables\/DataMan\/VACEngine.tcc/' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/VACEngine.tcc

pull the source from casacore again

casatables_impl/update.sh ../casacore/build

apply changes again

these changes are no longer necessary?

casatables_impl/casacore/casa/Arrays/Array2Math.cc
- memcpy (const_cast<Complex*>(carray.data()), static_cast<const void*>(rarray.data()), rarray.nelements() * sizeof(Float)); is now std::copy_n(rarray.data(), rarray.nelements(), reinterpret_cast<float*>(const_cast<std::complex<float>*>(carray.data()))); ?
casatables_impl/casacore/casa/System/ObjectID2.cc
- now has no casts ?

these changes were manually applied:

buffer sizes 1bf45d2
- casatables_impl/casacore/tables/Tables/ReadAsciiTable.cc
impl =
- casatables_impl/casacore/casa/Quanta/MV*.h
rubbl customization: never actually use any config files
- casatables_impl/casacore/casa/System/Aipsrc.cc
remove TAQL references 79fe0b2
- casatables_impl/casacore/tables/DataMan.h
- casatables_impl/casacore/tables/DataMan/{MSMIndColumn,DataManager}.cc
- casatables_impl/casacore/tables/Tables/{BaseTable,Table}.cc
- casatables_impl/casacore/tables/Tables/Table.cc
- casatables_impl/casacore/tables/Tables.h

safe to checkout from master:

git checkout master -- casatables_impl/casacore/casa/Containers/RecordDescRep.cc

and now, it takes several minutes just to print all the compiler errors, because of these cyclic dependencies

flowchart TB
classDef highlight fill:#f0f;
casa/Arrays.h --"includes"--> casa/Arrays/IPosition.h
casa/Arrays.h --"includes"--> casa/Arrays/Slicer.h
casa/Arrays.h --"includes"--> casa/Arrays/Slice.h
casa/Arrays.h --"includes"--> casa/Arrays/Array.h
casa/Arrays.h --"includes"--> casa/Arrays/Vector.h
class casa/Arrays/Vector.h highlight

casa/Arrays/Vector.h --"includes"--> casa/Arrays/Array.h
casa/Arrays/Array.h --"includes"--> casa/Arrays/ArrayBase.h
casa/Arrays/Array.h --"includes"--> casa/Arrays/ArrayError.h
casa/Arrays/Array.h --"includes"--> casa/Arrays/IPosition.h
casa/Arrays/ArrayBase.h --"includes"--> casa/Arrays/IPosition.h
casa/Arrays/ArrayError.h --"includes"--> casa/Arrays/IPosition.h
casa/Arrays/Slicer.h --"includes"--> casa/Arrays/IPosition.h

%% however
casa/Arrays/Slice.h -."should include".-> casa/Arrays/Vector.h
casa/Arrays/IPosition.h -."should include".-> casa/Arrays/Vector.h

%% casa/Arrays/IPosition.h --"includes"--> casa/Arrays/ArrayFwd.h
%% casa/Arrays.h --"includes"--> casa/Arrays/Matrix.h
%% casa/Arrays.h --"includes"--> casa/Arrays/Cube.h
%% casa/Arrays.h --"includes"--> casa/Arrays/ArrayIter.h
%% casa/Arrays.h --"includes"--> casa/Arrays/MatrixIter.h
%% casa/Arrays.h --"includes"--> casa/Arrays/VectorIter.h
%% casa/Arrays/ArrayPosIter.h --"includes"--> casa/Arrays/IPosition.h
%% casa/Arrays/ArrayMapping.h --"includes"--> casa/Arrays/IPosition.h
%% casa/Arrays/ArraySpecifier.h --"includes"--> casa/Arrays/IPosition.h
%% casa/Arrays/Array.h --"includes"--> casa/Arrays/Storage.h

casacore-3.5.0

casacore diff

d3v-null · 2024-05-17T06:48:21Z

finally got 3.5.0 tests passing! diff (from 3.3.0) here: d3v-null@db7a405

these are the only remaining compiler warnings:

    Blocking waiting for file lock on build directory
   Compiling rubbl_casatables_impl v0.0.0-dev.0 (/home/dev/src/rubbl/casatables_impl)
warning: rubbl_casatables_impl@0.0.0-dev.0: casacore/casa/OS/Memory.cc: In static member function ‘static size_t rubbl_casacore::Memory::allocatedMemoryInBytes()’:
warning: rubbl_casatables_impl@0.0.0-dev.0: casacore/casa/OS/Memory.cc:75:32: warning: ‘mallinfo mallinfo()’ is deprecated [-Wdeprecated-declarations]
warning: rubbl_casatables_impl@0.0.0-dev.0:    75 |    struct mallinfo m = mallinfo();
warning: rubbl_casatables_impl@0.0.0-dev.0:       |                        ~~~~~~~~^~
warning: rubbl_casatables_impl@0.0.0-dev.0: In file included from ./casacore/casa/OS/malloc.h:52,
warning: rubbl_casatables_impl@0.0.0-dev.0:                  from casacore/casa/OS/Memory.cc:37:
warning: rubbl_casatables_impl@0.0.0-dev.0: /usr/include/malloc.h:114:24: note: declared here
warning: rubbl_casatables_impl@0.0.0-dev.0:   114 | extern struct mallinfo mallinfo (void) __THROW __MALLOC_DEPRECATED;
warning: rubbl_casatables_impl@0.0.0-dev.0:       |                        ^~~~~~~~
warning: rubbl_casatables_impl@0.0.0-dev.0: casacore/casa/OS/Memory.cc: In static member function ‘static size_t rubbl_casacore::Memory::assignedMemoryInBytes()’:
warning: rubbl_casatables_impl@0.0.0-dev.0: casacore/casa/OS/Memory.cc:92:33: warning: ‘mallinfo mallinfo()’ is deprecated [-Wdeprecated-declarations]
warning: rubbl_casatables_impl@0.0.0-dev.0:    92 |     struct mallinfo m = mallinfo();
warning: rubbl_casatables_impl@0.0.0-dev.0:       |                         ~~~~~~~~^~
warning: rubbl_casatables_impl@0.0.0-dev.0: In file included from ./casacore/casa/OS/malloc.h:52,
warning: rubbl_casatables_impl@0.0.0-dev.0:                  from casacore/casa/OS/Memory.cc:37:
warning: rubbl_casatables_impl@0.0.0-dev.0: /usr/include/malloc.h:114:24: note: declared here
warning: rubbl_casatables_impl@0.0.0-dev.0:   114 | extern struct mallinfo mallinfo (void) __THROW __MALLOC_DEPRECATED;
warning: rubbl_casatables_impl@0.0.0-dev.0:       |                        ^~~~~~~~
   Compiling rubbl_casatables v0.0.0-dev.0 (/home/dev/src/rubbl/casatables)
warning: use of deprecated associated function `ndarray::impl_constructors::<impl ndarray::ArrayBase<S, D>>::uninitialized`: This method is hard to use correctly. Use `uninit` instead.
   --> casatables/src/lib.rs:363:27
    |
363 |         Ok(unsafe { Self::uninitialized(D::from_shape_slice(shape)?) })
    |                           ^^^^^^^^^^^^^
    |
    = note: `#[warn(deprecated)]` on by default

warning: dereferencing a null pointer
   --> casatables/src/glue.rs:110:19
    |
110 |         unsafe { &(*(::std::ptr::null::<StringBridge>())).data as *const _ as usize },
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this code causes undefined behavior when executed
    |
    = note: `#[warn(deref_nullptr)]` on by default

warning: dereferencing a null pointer
   --> casatables/src/glue.rs:120:19
    |
120 |         unsafe { &(*(::std::ptr::null::<StringBridge>())).n_bytes as *const _ as usize },
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^ this code causes undefined behavior when executed

warning: dereferencing a null pointer
   --> casatables/src/glue.rs:148:19
    |
148 |         unsafe { &(*(::std::ptr::null::<ExcInfo>())).message as *const _ as usize },
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this code causes undefined behavior when executed

steps to go from v3.3.0 to 3.5.0

these are the steps I applied from the v3.3.0 branch I mentioned in the last comment ( i gave up on 3.4.0. I probably just made a simple mistake and got lucky this time around).

# 3.3.0 -> 3.4.0

# move casa/Arrays/ArrayIO → casa/IO/ArrayIO
mv casatables_impl/casacore/casa/Arrays/ArrayIO.{h,tcc} casatables_impl/casacore/casa/IO/
sed -i 's!Arrays/ArrayIO!IO/ArrayIO!' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/IO/ArrayIO.{h,tcc}
# move casa/Arrays/IPosition2 → casa/IO/IPositionIO
mv casatables_impl/casacore/casa/Arrays/IPosition2.cc casatables_impl/casacore/casa/IO/IPositionIO.cc
sed -i 's!Arrays/IPosition2!IO/IPositionIO!' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/IO/IPositionIO.cc
# move casa/Arrays/LogiArrayFwd → casa/Arrays/ArrayFwd
mv casatables_impl/casacore/casa/Arrays/LogiArrayFwd.h casatables_impl/casacore/casa/Arrays/ArrayFwd.h
sed -i 's!Arrays/LogiArrayFwd!Arrays/ArrayFwd!' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/Arrays/ArrayFwd.h
# remove casa/BasicSL/RegexBase, casa/Utilities/cregex
rm casatables_impl/casacore/casa/BasicSL/RegexBase.{cc,h}
git stage casatables_impl/casacore/casa/BasicSL/RegexBase.{cc,h}
sed -i '/RegexBase/d' casatables_impl/build.rs
rm casatables_impl/casacore/casa/Utilities/cregex.{cc,h}
git stage casatables_impl/casacore/casa/Utilities/cregex.{cc,h}
sed -i '/cregex/d' casatables_impl/build.rs
# move tables/Tables/ArrColData.tcc → tables/Tables/ArrColData.cc
mv casatables_impl/casacore/tables/Tables/ArrColData.tcc casatables_impl/casacore/tables/Tables/ArrColData.cc
sed -i 's!ArrColData.tcc!ArrColData.cc!' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/ArrColData.cc
# add casa/Arrays/Storage
touch casatables_impl/casacore/casa/Arrays/Storage.h
sed -i '/casacore\/casa\/Arrays\/Slicer.h/a \    "casacore\/casa\/Arrays\/Storage.h",' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/Arrays/Storage.h

# add casa/Arrays/ArrayStr
touch casatables_impl/casacore/casa/Arrays/ArrayStr.{h,tcc}
sed -i '/casacore\/casa\/Arrays\/ArrayPosIter.h/a \    "casacore/casa/Arrays/ArrayStr.h",' casatables_impl/build.rs
sed -i '/casacore\/casa\/Arrays\/ArrayPosIter.h/a \    "casacore/casa/Arrays/ArrayStr.tcc",' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/Arrays/ArrayStr.{h,tcc}
# add casa/Arrays/Memory
touch casatables_impl/casacore/casa/Arrays/Memory.h
sed -i '/casacore\/casa\/Arrays\/Matrix.tcc/a \    "casacore/casa/Arrays/Memory.h",' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/Arrays/Memory.h
# add casa/Arrays/ElementFunctions.h
touch casatables_impl/casacore/casa/Arrays/ElementFunctions.h
sed -i '/casacore\/casa\/Arrays\/Cube.tcc/a \    "casacore/casa/Arrays/ElementFunctions.h",' casatables_impl/build.rs
git stage casatables_impl/casacore/casa/Arrays/ElementFunctions.h

# add tables/Tables/ArrayColumnBase.cc
touch casatables_impl/casacore/tables/Tables/ArrayColumnBase.cc
sed -i '/casacore\/tables\/Tables\/ArrColDesc_tmpl.cc/a \    "casacore/tables/Tables/ArrayColumnBase.cc",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/ArrayColumnBase.cc
# add tables/Tables/ArrColData.cc
touch casatables_impl/casacore/tables/Tables/ArrColData.cc
sed -i '/casacore\/tables\/Tables\/ArrayColumnBase.cc/a \    "casacore/tables/Tables/ArrColData.cc",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/ArrColData.cc
# add tables/Tables/ArrColDesc.cc
touch casatables_impl/casacore/tables/Tables/ArrColDesc.cc
sed -i '/casacore\/tables\/Tables\/ArrayColumnBase.cc/a \    "casacore/tables/Tables/ArrColDesc.cc",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/ArrColDesc.cc
# add tables/Tables/RowNumbers.cc
touch casatables_impl/casacore/tables/Tables/RowNumbers.cc
sed -i '/casacore\/tables\/Tables\/RowCopier.cc/a \    "casacore/tables/Tables/RowNumbers.cc",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/RowNumbers.cc
# add tables/DataMan/DataManagerColumn.cc
touch casatables_impl/casacore/tables/DataMan/DataManagerColumn.cc
sed -i '/casacore\/tables\/DataMan\/DataManager.cc/a \    "casacore/tables/DataMan/DataManagerColumn.cc",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/DataManagerColumn.cc
# add tables/DataMan/StManColumnBase.cc
touch casatables_impl/casacore/tables/DataMan/StManColumnBase.cc
sed -i '/casacore\/tables\/DataMan\/StManColumn.cc/a \    "casacore/tables/DataMan/StManColumnBase.cc",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/StManColumnBase.cc
# add tables/DataMan/VirtArrCol.cc
touch casatables_impl/casacore/tables/DataMan/VirtArrCol.cc
sed -i '/casacore\/tables\/DataMan\/VirtColEng.cc/a \    "casacore/tables/DataMan/VirtArrCol.cc",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/VirtArrCol.cc
# add tables/DataMan/VirtScaCol.cc
touch casatables_impl/casacore/tables/DataMan/VirtScaCol.cc
sed -i '/casacore\/tables\/DataMan\/VirtColEng.cc/a \    "casacore/tables/DataMan/VirtScaCol.cc",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/VirtScaCol.cc
# add tables/Tables/ArrayColumnBase.h
touch casatables_impl/casacore/tables/Tables/ArrayColumnBase.h
sed -i '/casacore\/tables\/Tables\/ArrayColumn.h/a \    "casacore/tables/Tables/ArrayColumnBase.h",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/ArrayColumnBase.h
# add tables/Tables/RowNumbers.h
touch casatables_impl/casacore/tables/Tables/RowNumbers.h
sed -i '/casacore\/tables\/Tables\/RowCopier.h/a \    "casacore/tables/Tables/RowNumbers.h",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/Tables/RowNumbers.h
# add tables/DataMan/DataManagerColumn.h
touch casatables_impl/casacore/tables/DataMan/DataManagerColumn.h
sed -i '/casacore\/tables\/DataMan\/DataManager.h/a \    "casacore/tables/DataMan/DataManagerColumn.h",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/DataManagerColumn.h
# add tables/DataMan/StManColumnBase.h
touch casatables_impl/casacore/tables/DataMan/StManColumnBase.h
sed -i '/casacore\/tables\/DataMan\/StManColumn.h/a \    "casacore/tables/DataMan/StManColumnBase.h",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/StManColumnBase.h
# add tables/DataMan/VACEngine.h
touch casatables_impl/casacore/tables/DataMan/VACEngine.h
sed -i '/casacore\/tables\/DataMan\/VSCEngine.tcc/a \    "casacore/tables/DataMan/VACEngine.h",' casatables_impl/build.rs
git stage casatables_impl/casacore/tables/DataMan/VACEngine.h
# add tables/DataMan/VACEngine.tcc
touch casatables_impl/casacore/tables/DataMan/VACEngine.tcc
sed -i '/casacore\/tables\/DataMan\/VSCEngine.tcc/a \    "casacore/tables/DataMan/VACEngine.tcc",' casatables_impl/build.rs

# 3.4.0 -> 3.5.0
# delete casa/Utilities/Register.{tcc,h}
git rm casatables_impl/casacore/casa/Utilities/Register.{tcc,h}
sed -i '!casacore/casa/Utilities/Register.tcc!d' casatables_impl/build.rs
sed -i '!casacore/casa/Utilities/Register.h!d' casatables_impl/build.rs
# delete casa/Utilities/RegSequence.{cc,h}
git rm casatables_impl/casacore/casa/Utilities/RegSequence.{cc,h}
sed -i '!casacore/casa/Utilities/RegSequence.cc!d' casatables_impl/build.rs
sed -i '!casacore/casa/Utilities/RegSequence.h!d' casatables_impl/build.rs
# delete casa/OS/Mutex.cc
git rm casatables_impl/casacore/casa/OS/Mutex.cc
sed -i '!casacore/casa/OS/Mutex.cc!d' casatables_impl/build.rs
# add casacore/casa/Utilities/Sort.tcc
touch casatables_impl/casacore/casa/Utilities/Sort.tcc
sed -i '/casacore\/casa\/Utilities\/Sort.h/a \    "casacore/casa/Utilities/Sort.tcc",' casatables_impl/build.rs
git add casatables_impl/casacore/casa/Utilities/Sort.tcc
# add casacore/tables/Tables/TableUtil.h
touch casatables_impl/casacore/tables/Tables/TableUtil.h
sed -i '/casacore\/tables\/Tables\/TableTrace.h/a \    "casacore/tables/Tables/TableUtil.h",' casatables_impl/build.rs
git add casatables_impl/casacore/tables/Tables/TableUtil.h
casatables_impl/update.sh ../casacore/build

re-apply rubbl changes

no longer necessary

casa/Arrays/Array2Math.cc

// in v3.3.0 it was
memcpy (const_cast<Complex*>(carray.data()), rarray.data(), rarray.nelements() * sizeof(Float));
// in rubbl:master it is
memcpy (const_cast<Complex*>(carray.data()), static_cast<const void*>(rarray.data()), rarray.nelements() * sizeof(Float));
// in v3.4.0 it is
std::copy_n(rarray.data(), rarray.nelements(), reinterpret_cast<float*>(const_cast<std::complex<float>*>(carray.data())));

safe to checkout

git checkout master -- casatables_impl/casacore/casa/System/ObjectID2.cc

patch manually

# buffer sizes
sed -i 's!char strc\[16\];      // should be plenty large...!char strc[32];!' casatables_impl/casacore/casa/Containers/RecordDescRep.cc
sed -i 's!char name\[24\];!char name[32];!' casatables_impl/casacore/tables/Tables/ReadAsciiTable.cc
# impl =
sed -i '/typ((MVAngle::formatTypes)/a \    Format& operator=(const Format&) = default;' casatables_impl/casacore/casa/Quanta/MVAngle.h
sed -i '/typ((MVTime::formatTypes)/a \    Format& operator=(const Format&) = default;' casatables_impl/casacore/casa/Quanta/MVTime.h
# impl default
sed -i '/# Operators/a \  MVBaseline(const MVBaseline&) = default;' casatables_impl/casacore/casa/Quanta/MVBaseline.h
sed -i '/# Operators/a \  MVDirection(const MVDirection&) = default;' casatables_impl/casacore/casa/Quanta/MVDirection.h
sed -i '/# Operators/a \  MVEarthMagnetic(const MVEarthMagnetic&) = default;' casatables_impl/casacore/casa/Quanta/MVEarthMagnetic.h
sed -i '/# Operators/a \  MVuvw(const MVuvw&) = default;' casatables_impl/casacore/casa/Quanta/MVuvw.h
# remove TAQL references https://github.com/pkgw/rubbl/commit/79fe0b245d0537b4fdf837ef6cebd13be8661abe
sed -i '/VirtualTaQLColumn/d' casatables_impl/casacore/tables/DataMan.h
sed -i '/VirtualTaQLColumn/d' casatables_impl/casacore/tables/DataMan/DataManager.cc
sed -i '/TaQL/d' casatables_impl/casacore/tables/Tables.h
sed -i '/ExprNode/d' casatables_impl/casacore/tables/Tables/BaseTable.cc
sed -i '/BaseTable::select (const TableExprNode/,/return resultTable.transfer();/d' casatables_impl/casacore/tables/Tables/BaseTable.cc
sed -i '/Do the row selection./,/}/d' casatables_impl/casacore/tables/Tables/BaseTable.cc
# don't throw DataManInternalError
sed -i '/if (data_p \!= 0) {/,/}/d' casatables_impl/casacore/tables/DataMan/MSMIndColumn.cc
# !! plus one more line

# in casatables_impl/casacore/tables/Tables/Table.cc, remove ExprDerNode.h and ExprNode.h
sed -i '/ExprDerNode.h/d' casatables_impl/casacore/tables/Tables/Table.cc
sed -i '/ExprNode.h/d' casatables_impl/casacore/tables/Tables/Table.cc
sed -i '/# Create an expression node to handle a keyword./,/# Select rows based on row numbers./d' casatables_impl/casacore/tables/Tables/Table.cc

it even passes on macos arm64 with hundreds of warnings!

macos-14-x86.txt

So... now that I've been able to show it's possible, what's next?

Sorry for all the spam! Hopefully the spoiler tags give you the tl;dr

d3v-null · 2024-07-10T00:47:24Z

Hey @pkgw , Is there anything I can do to help progress this issue? I know there's a lot here so I'll just summarize.

I hope I have adequately demonstrated the changes needed to update the casacore source between each minor version (except 3.4.0).

current to 3.2.1 - trivial
3.2.1 to 3.3.0 - trivial
3.3.0 to 3.5.0 - significant changes

The question is: what needs to happen next?

This is my understanding, but I wanted to hear your thoughts:

The vendor-casacore branch points to a version of casatables_impl that contains unmodified casacore source code files (before the casatables package existed).
The commits between vendor-casacore and main include changes to casatables_impl along with the entire history of the casatables package.
Currently, update.sh does not handle when a file in casacore has been moved, or a new file is needed, so a bit of manual handling is needed sometimes.

So to update casacore versions, one would need to:

checkout vendor-casacore
import raw source code from a newer version of casacore (update.sh plus some manual handling for v3.5.0)
(for v3.5.0) make changes to build.rs
rebase main onto vendor-casacore, rewriting history in some cases to modify commits
get tests passing in main
do a release?

I can't see a way around this that doesn't involve rewriting history and force pushing to main, which I would need special permissions to do on this repo.

If this is something that you don't have time to do, I totally understand. I may however need to start working on my own fork. If so, do you have any tips on how I might use your cranko-based release process on my own fork ? would it just be a matter of renaming all the packages to be mwa_rubbl_* ?

Thanks again.

pkgw · 2024-08-09T20:33:44Z

I am finally working on this. Sorry for taking so long to follow up with you.

PR #389 merges in your work.

One important correction/clarification to the checklist you wrote in your latest comment (from a month ago, sigh): once vendor-casacore is updated, it is not necessary to rebase main onto it.

Instead, the model is to merge the updated vendor-casacore branch into main. The resulting history graph looks like, uh, this, if it is at all helpful:

* main
|
* <-------\
|          \
*          *  vendor-casacore
|          |
*          |
|          |
*          *
|          |
* <------\ |
|         \|
*          *
|          |
[ .. etc .. ]

The point is, you can merge vendor-casacore into main multiple times — each subsequent merge only has to bring in the contents of the new commits since the prior merge. The nice thing about this approach is that if there are hacks to the source code that happen in main, they do not need to be re-applied unless the updates to vendor-casacore invalidate them.

While I was writing this, #389 built and I merged it in. I will take a look at whether there are any other touchups needed. In particular, it is probably time to deal with the warning about uninitialized memory in ndarray, which may necessitate an API change.

d3v-null · 2024-08-12T02:43:55Z

Thanks Peter!

rubbl main is working great for me.

I will take a look at whether there are any other touchups needed

Other than the sprintf warnings, the only other warnings I'm seeing are

casacore/casa/OS/PrecTimer.cc:94:2: warning: partially supported architecture [-W#warnings]
#warning partially supported architecture
 ^
 
src/glue.cc:42:5: warning: 'bridge_string' has C-linkage specified, but returns user-defined type 'rubbl_casacore::String' which is incompatible with C [-Wreturn-type-c-linkage]
    bridge_string(const StringBridge &input)
    ^
src/glue.cc:68:5: warning: 'bridge_string_array' has C-linkage specified, but returns user-defined type 'rubbl_casacore::Array<rubbl_casacore::String>' which is incompatible with C [-Wreturn-type-c-linkage]
    bridge_string_array(const StringBridge *source, const casacore::IPosition &shape)
    ^

use of deprecated associated function `ndarray::impl_constructors::<impl ndarray::ArrayBase<S, D>>::uninitialized`: This method is hard to use correctly. Use `uninit` instead.
   --> casatables/src/lib.rs:363:27
    |
363 |         Ok(unsafe { Self::uninitialized(D::from_shape_slice(shape)?) })
    |                           ^^^^^^^^^^^^^
    |
    = note: `#[warn(deprecated)]` on by default

edit: once you've got a handle on the ndarray thing, It'd be great if we could do a release.

Thanks!

pkgw · 2024-08-13T20:45:59Z

New releases are out!

https://dev.azure.com/pkgw/Misc/_build/results?buildId=1491&view=results
https://github.com/pkgw/rubbl/releases/tag/rubbl_casatables%400.8.0
https://doi.org/10.5281/zenodo.13315460

d3v-null mentioned this issue Apr 3, 2024

Release pre-compiled packages for macOS M1/M2/M3 MWATelescope/mwa_hyperbeam#7

Closed

d3v-null mentioned this issue Jul 30, 2024

Fix non-x86 CPUs not compiling. MWATelescope/mwa_hyperdrive#34

Merged

pkgw mentioned this issue Aug 9, 2024

Bring up mainline to casacore 3.5.0 #389

Merged

d3v-null closed this as completed Aug 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

can't compile casatables_impl on arm64: narrowing conversion from 'int' to 'char' #345

can't compile casatables_impl on arm64: narrowing conversion from 'int' to 'char' #345

d3v-null commented Mar 9, 2024 •

edited

Loading

d3v-null commented Mar 9, 2024 •

edited

Loading

pkgw commented Mar 12, 2024

d3v-null commented Mar 12, 2024

pkgw commented Mar 12, 2024

d3v-null commented Apr 16, 2024 •

edited

Loading

d3v-null commented May 16, 2024 •

edited

Loading

casacore-3.2.1

casacore-3.3.0

casacore-3.4.0

casacore-3.5.0

d3v-null commented May 17, 2024 •

edited

Loading

re-apply rubbl changes

no longer necessary

safe to checkout

patch manually

d3v-null commented Jul 10, 2024

pkgw commented Aug 9, 2024

d3v-null commented Aug 12, 2024 •

edited

Loading

pkgw commented Aug 13, 2024 •

edited

Loading

can't compile casatables_impl on arm64: narrowing conversion from 'int' to 'char' #345

can't compile casatables_impl on arm64: narrowing conversion from 'int' to 'char' #345

Comments

d3v-null commented Mar 9, 2024 • edited Loading

d3v-null commented Mar 9, 2024 • edited Loading

pkgw commented Mar 12, 2024

d3v-null commented Mar 12, 2024

pkgw commented Mar 12, 2024

d3v-null commented Apr 16, 2024 • edited Loading

d3v-null commented May 16, 2024 • edited Loading

tl;dr

casacore-3.2.1

casacore-3.3.0

casacore-3.4.0

casacore-3.5.0

d3v-null commented May 17, 2024 • edited Loading

re-apply rubbl changes

no longer necessary

safe to checkout

patch manually

d3v-null commented Jul 10, 2024

pkgw commented Aug 9, 2024

d3v-null commented Aug 12, 2024 • edited Loading

pkgw commented Aug 13, 2024 • edited Loading

d3v-null commented Mar 9, 2024 •

edited

Loading

d3v-null commented Mar 9, 2024 •

edited

Loading

d3v-null commented Apr 16, 2024 •

edited

Loading

d3v-null commented May 16, 2024 •

edited

Loading

d3v-null commented May 17, 2024 •

edited

Loading

d3v-null commented Aug 12, 2024 •

edited

Loading

pkgw commented Aug 13, 2024 •

edited

Loading