Replacing long int type with int64_t

[chazeon]

Following the discussion in #3657 this pull request addresses the usage of long or long int by replacing them with int64_t in multiple instances. This change aims to enhance code compatibility across different platforms and improve code clarity.

The int64_t type is a feature introduced in C++11, defined in the <cstdint> header. Due to historical reasons, the compilation behavior of int64_t is platform- and system-specific. On Linux, int64_t is compiled to long, whereas on macOS, it's compiled to long long.

In relevant codebases such as PyTorch and TensorFlow, int64_t is preferred over explicit declarations of long or long long. Consequently, for precompiled libraries, on Linux, symbols are defined exclusively to long, while on macOS, symbols are defined exclusively based on long long.

For these reasons, data_ptr<long int>() is unable to compile on macOS.

References

• https://stackoverflow.com/questions/67584843/pytorch-tensordata-ptrlong-long-not-working-on-linux
• https://github.com/llvm/llvm-project/blob/c7910ee1f0af64501bf068cdfec154ea359ff832/clang/test/Preprocessor/init.c

Examples

Example 1

For the code used here, torch::from_blob, is defined using

inline at::Tensor from_blob(
    void* data,
    at::IntArrayRef sizes,
    const at::TensorOptions& options = at::TensorOptions()) {

where IntArrayRef is defined as

using IntArrayRef = c10::ArrayRef<int64_t>;

Example 2

Dumping the symbols in libtorch_cpu.dylib on macOS

-> % nm -gU libtorch_cpu.dylib | llvm-cxxfilt | grep TensorBase | grep ::data_ptr
0000000002153398 T c10::Float8_e5m2* at::TensorBase::data_ptr<c10::Float8_e5m2>() const
0000000002153524 T c10::Float8_e4m3fn* at::TensorBase::data_ptr<c10::Float8_e4m3fn>() const
0000000002152738 T c10::Half* at::TensorBase::data_ptr<c10::Half>() const
00000000021536b0 T c10::qint8* at::TensorBase::data_ptr<c10::qint8>() const
00000000021539c8 T c10::qint32* at::TensorBase::data_ptr<c10::qint32>() const
000000000215383c T c10::quint8* at::TensorBase::data_ptr<c10::quint8>() const
0000000002152bdc T c10::complex<c10::Half>* at::TensorBase::data_ptr<c10::complex<c10::Half>>() const
0000000002152ef4 T c10::complex<double>* at::TensorBase::data_ptr<c10::complex<double>>() const
0000000002152d68 T c10::complex<float>* at::TensorBase::data_ptr<c10::complex<float>>() const
000000000215320c T c10::BFloat16* at::TensorBase::data_ptr<c10::BFloat16>() const
0000000002153ce0 T c10::quint2x4* at::TensorBase::data_ptr<c10::quint2x4>() const
0000000002153b54 T c10::quint4x2* at::TensorBase::data_ptr<c10::quint4x2>() const
0000000002152108 T signed char* at::TensorBase::data_ptr<signed char>() const
0000000002153080 T bool* at::TensorBase::data_ptr<bool>() const
0000000002152a50 T double* at::TensorBase::data_ptr<double>() const
00000000021528c4 T float* at::TensorBase::data_ptr<float>() const
0000000002151f7c T unsigned char* at::TensorBase::data_ptr<unsigned char>() const
0000000002152420 T int* at::TensorBase::data_ptr<int>() const
0000000002152294 T short* at::TensorBase::data_ptr<short>() const
00000000021525ac T long long* at::TensorBase::data_ptr<long long>() const

dumping symbols in libtorch_cpu.dylib on Linux

-> % nm -gU libtorch_cpu.so | c++filt | grep TensorBase | grep ::data_ptr 
00000000031ec0d0 T c10::Float8_e5m2* at::TensorBase::data_ptr<c10::Float8_e5m2>() const
00000000031ec2f0 T c10::Float8_e4m3fn* at::TensorBase::data_ptr<c10::Float8_e4m3fn>() const
00000000031ec730 T c10::Float8_e4m3fnuz* at::TensorBase::data_ptr<c10::Float8_e4m3fnuz>() const
00000000031ec510 T c10::Float8_e5m2fnuz* at::TensorBase::data_ptr<c10::Float8_e5m2fnuz>() const
00000000031eb030 T c10::Half* at::TensorBase::data_ptr<c10::Half>() const
00000000031ec950 T c10::qint8* at::TensorBase::data_ptr<c10::qint8>() const
00000000031ecd80 T c10::qint32* at::TensorBase::data_ptr<c10::qint32>() const
00000000031ecb70 T c10::quint8* at::TensorBase::data_ptr<c10::quint8>() const
00000000031eb660 T c10::complex<c10::Half>* at::TensorBase::data_ptr<c10::complex<c10::Half> >() const
00000000031eba80 T c10::complex<double>* at::TensorBase::data_ptr<c10::complex<double> >() const
00000000031eb870 T c10::complex<float>* at::TensorBase::data_ptr<c10::complex<float> >() const
00000000031ebeb0 T c10::BFloat16* at::TensorBase::data_ptr<c10::BFloat16>() const
00000000031ed1c0 T c10::quint2x4* at::TensorBase::data_ptr<c10::quint2x4>() const
00000000031ecfa0 T c10::quint4x2* at::TensorBase::data_ptr<c10::quint4x2>() const
00000000031ea7f0 T signed char* at::TensorBase::data_ptr<signed char>() const
00000000031ebca0 T bool* at::TensorBase::data_ptr<bool>() const
00000000031eb450 T double* at::TensorBase::data_ptr<double>() const
00000000031eb240 T float* at::TensorBase::data_ptr<float>() const
00000000031ea5d0 T unsigned char* at::TensorBase::data_ptr<unsigned char>() const
00000000031eac10 T int* at::TensorBase::data_ptr<int>() const
00000000031ed5e0 T unsigned int* at::TensorBase::data_ptr<unsigned int>() const
00000000031eae20 T long* at::TensorBase::data_ptr<long>() const
00000000031ed7f0 T unsigned long* at::TensorBase::data_ptr<unsigned long>() const
00000000031eaa00 T short* at::TensorBase::data_ptr<short>() const
00000000031ed3d0 T unsigned short* at::TensorBase::data_ptr<unsigned short>() const

Summary by CodeRabbit

• Refactor
  • Improved data type consistency across various components for handling larger data sizes more reliably.

[coderabbitai]

Walkthrough

Walkthrough

The recent updates across various source files primarily focus on enhancing data type consistency for array and index size handling. Changes include shifting from long int and unsigned long to int64_t and uint64_t respectively. These modifications ensure better compatibility and precision in memory and index calculations, particularly useful in environments with large datasets or high computational demands.

Changes

File Path	Change Summary
.../src/DeepPotPT.cc	Updated type casting for array sizes from long int to int64_t in the compute function.
.../src/DeepPotTF.cc	Changed index calculation casting from unsigned long to uint64_t in the tile_fparam_aparam function.
.../op/pt/comm.cc	Updated data types for communicator handling from long int to int64_t in the unpack_communicator method.

---

Recent Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between a16becc and de5ff49.

Files selected for processing (1)

• source/op/pt/comm.cc (3 hunks)

Files skipped from review as they are similar to previous changes (1)

• source/op/pt/comm.cc

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.18%. Comparing base (ebd809b) to head (de5ff49).
Report is 1 commits behind head on devel.

Additional details and impacted files

@@            Coverage Diff             @@
##            devel    #3739      +/-   ##
==========================================
- Coverage   82.19%   82.18%   -0.01%     
==========================================
  Files         513      513              
  Lines       47642    47642              
  Branches     2980     2980              
==========================================
- Hits        39159    39156       -3     
- Misses       7572     7575       +3     
  Partials      911      911              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[njzjz]

defined in the header

should it be included?

[chazeon]

After giving it some thought, because this observed behavior with libtorch_cpu.so, and because the compiler didn't complain, seems like <cstdint> has already being included somewhere along the chain of includes, although this wasn't clearly documented. Still, to be on the safe side, I think it's a good practice for me to explicitly include <cstdint> here. Being more explicit never hurts. Additionally, I've decided to use std::int64_t instead of just int64_t, even though current compilers typically don't complain about it. I will push a new commit shortly.