Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .buildkite/pipelines/build_linux.json.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"cpu": "6",
"ephemeralStorage": "20G",
"memory": "64G",
"image": os.getenv("DOCKER_IMAGE", "docker.elastic.co/ml-dev/ml-linux-build:32")
"image": os.getenv("DOCKER_IMAGE", "docker.elastic.co/ml-dev/ml-linux-build:32a")
},
"aarch64": {
"provider": "aws",
Expand Down Expand Up @@ -101,7 +101,7 @@ def main(args):
"cpu": "6",
"ephemeralStorage": "20G",
"memory": "64G",
"image": "docker.elastic.co/ml-dev/ml-linux-aarch64-cross-build:15"
"image": "docker.elastic.co/ml-dev/ml-linux-aarch64-cross-build:15a"
},
"commands": [
".buildkite/scripts/steps/build_and_test.sh"
Expand Down
2 changes: 1 addition & 1 deletion 3rd_party/licenses/pytorch-INFO.csv
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
name,version,revision,url,license,copyright,sourceURL
PyTorch,2.5.1,a8d6afb511a69687bbb2b7e88a3cf67917e1697e,https://pytorch.org,BSD-3-Clause,,
PyTorch,2.7.1,e2d141dbde55c2a4370fac5165b0561b6af4798b,https://pytorch.org,BSD-3-Clause,,
6 changes: 3 additions & 3 deletions build-setup/linux.md
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,7 @@ Then copy the shared libraries to the system directory:
(cd /opt/intel/oneapi/mkl/2024.0 && tar cf - lib) | (cd /usr/local/gcc103 && sudo tar xvf -)
```

### PyTorch 2.5.1
### PyTorch 2.7.1

(This step requires a reasonable amount of memory. It failed on a machine with 8GB of RAM. It succeeded on a 16GB machine. You can specify the number of parallel jobs using environment variable MAX_JOBS. Lower number of jobs will reduce memory usage.)

Expand All @@ -348,7 +348,7 @@ sudo /usr/local/gcc103/bin/python3.10 -m pip install install numpy pyyaml setupt
Then obtain the PyTorch code:

```
git clone --depth=1 --branch=v2.5.1 git@github.com:pytorch/pytorch.git
git clone --depth=1 --branch=v2.7.1 git@github.com:pytorch/pytorch.git
cd pytorch
git submodule sync
git submodule update --init --recursive
Expand Down Expand Up @@ -383,7 +383,7 @@ export USE_MKLDNN=ON
export USE_QNNPACK=OFF
export USE_PYTORCH_QNNPACK=OFF
[ $(uname -m) = x86_64 ] && export USE_XNNPACK=OFF
export PYTORCH_BUILD_VERSION=2.5.1
export PYTORCH_BUILD_VERSION=2.7.1
export PYTORCH_BUILD_NUMBER=1
/usr/local/gcc103/bin/python3.10 setup.py install
```
Expand Down
6 changes: 3 additions & 3 deletions build-setup/macos.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ Download the graphical installer for Python 3.10.9 from <https://www.python.org/

Install using all the default options. When the installer completes a Finder window pops up. Double click the `Install Certificates.command` file in this folder to install the SSL certificates Python needs.

### PyTorch 2.5.1
### PyTorch 2.7.1

PyTorch requires that certain Python modules are installed. To install them:

Expand All @@ -138,7 +138,7 @@ sudo /Library/Frameworks/Python.framework/Versions/3.10/bin/pip3.10 install nump
Then obtain the PyTorch code:

```
git clone --depth=1 --branch=v2.5.1 https://github.com/pytorch/pytorch.git
git clone --depth=1 --branch=v2.7.1 https://github.com/pytorch/pytorch.git
cd pytorch
git submodule sync
git submodule update --init --recursive
Expand Down Expand Up @@ -252,7 +252,7 @@ export DNNL_TARGET_ARCH=AARCH64
export USE_MKLDNN=ON
export USE_QNNPACK=OFF
export USE_PYTORCH_QNNPACK=OFF
export PYTORCH_BUILD_VERSION=2.5.1
export PYTORCH_BUILD_VERSION=2.7.1
export PYTORCH_BUILD_NUMBER=1
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 setup.py install
```
Expand Down
6 changes: 3 additions & 3 deletions build-setup/windows.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ On the "Advanced Options" screen, check "Install for all users" and "Add Python

For the time being, do not take advantage of the option on the final installer screen to reconfigure the machine to allow paths longer than 260 characters. We still support Windows versions that do not have this option.

### PyTorch 2.5.1
### PyTorch 2.7.1

(This step requires a lot of memory. It failed on a machine with 12GB of RAM. It just about fitted on a 20GB machine. 32GB RAM is recommended.)

Expand All @@ -209,7 +209,7 @@ Next, in a Git bash shell run:

```
cd /c/tools
git clone --depth=1 --branch=v2.5.1 https://github.com/pytorch/pytorch.git
git clone --depth=1 --branch=v2.7.1 https://github.com/pytorch/pytorch.git
cd pytorch
git submodule sync
git submodule update --init --recursive
Expand Down Expand Up @@ -265,7 +265,7 @@ set USE_QNNPACK=OFF
set USE_PYTORCH_QNNPACK=OFF
set USE_XNNPACK=OFF
set MSVC_Z7_OVERRIDE=OFF
set PYTORCH_BUILD_VERSION=2.5.1
set PYTORCH_BUILD_VERSION=2.7.1
set PYTORCH_BUILD_NUMBER=1
python setup.py install
```
Expand Down
8 changes: 4 additions & 4 deletions dev-tools/docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,15 +36,15 @@ required to build the machine learning C++ code dependencies:
2. Change the Dockerfile and build a new image to be used for subsequent builds on this branch.
3. Update the version to be used for builds in docker files that refer to it.

### Depends on: ml-linux-build:32
### Depends on: ml-linux-build:32a

### Build script: dev-tools/docker/build_linux_dependency_builder_image.sh



## Repository: ml-linux-build

### Latest version: 32
### Latest version: 32a

### Comments
A Docker image that can be used to compile the machine learning
Expand All @@ -63,7 +63,7 @@ used for subsequent builds on this branch.

## Repository: ml-linux-aarch64-cross-build

### Latest version: 15
### Latest version: 15a

### Comments
A Docker image that can be used to compile the machine learning
Expand All @@ -82,7 +82,7 @@ used for subsequent builds on this branch.

## Repository: ml-linux-aarch64-native-build

### Latest version: 15
### Latest version: 15a

### Comments
A Docker image that can be used to compile the machine learning
Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/build_linux_aarch64_cross_build_image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
HOST=docker.elastic.co
ACCOUNT=ml-dev
REPOSITORY=ml-linux-aarch64-cross-build
VERSION=15
VERSION=15a

set -e

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/build_linux_aarch64_native_build_image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ sleep 5
HOST=docker.elastic.co
ACCOUNT=ml-dev
REPOSITORY=ml-linux-aarch64-native-build
VERSION=13
VERSION=15a

set -e

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/build_linux_build_image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ sleep 5
HOST=docker.elastic.co
ACCOUNT=ml-dev
REPOSITORY=ml-linux-build
VERSION=30
VERSION=32a

set -e

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_aarch64_cross_builder/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-cross-build:15
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-cross-build:15a

MAINTAINER David Roberts <dave.roberts@elastic.co>

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_aarch64_cross_image/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ RUN \
RUN \
mkdir -p /usr/local/sysroot-aarch64-linux-gnu/usr && \
cd /usr/local/sysroot-aarch64-linux-gnu/usr && \
wget --quiet -O - https://s3-eu-west-2.amazonaws.com/ml-cpp-artifacts/dependencies/usr-aarch64-linux-gnu-15.tar.bz2 | tar jxf - && \
wget --quiet -O - https://s3-eu-west-2.amazonaws.com/ml-cpp-artifacts/dependencies/usr-aarch64-linux-gnu-15a.tar.bz2 | tar jxf - && \
cd .. && \
ln -s usr/lib lib && \
ln -s usr/lib64 lib64
Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_aarch64_native_builder/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-native-build:15
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-native-build:15a

MAINTAINER David Roberts <dave.roberts@elastic.co>

Expand Down
6 changes: 3 additions & 3 deletions dev-tools/docker/linux_aarch64_native_image/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ ARG build_dir=/usr/src
# Build gcc 10.3
RUN \
cd ${build_dir} && \
wget --quiet -O - http://ftpmirror.gnu.org/gcc/gcc-10.3.0/gcc-10.3.0.tar.gz | tar zxf - && \
wget --quiet -O - https://mirrorservice.org/sites/sourceware.org/pub/gcc/releases/gcc-10.3.0/gcc-10.3.0.tar.gz | tar zxf - && \
cd gcc-10.3.0 && \
contrib/download_prerequisites && \
sed -i -e 's/$(SHLIB_LDFLAGS)/-Wl,-z,relro -Wl,-z,now $(SHLIB_LDFLAGS)/' libgcc/config/t-slibgcc && \
Expand Down Expand Up @@ -141,7 +141,7 @@ RUN \
# If the PyTorch branch is changed also update PYTORCH_BUILD_VERSION
RUN \
cd ${build_dir} && \
git -c advice.detachedHead=false clone --depth=1 --branch=v2.5.1 https://github.com/pytorch/pytorch.git && \
git -c advice.detachedHead=false clone --depth=1 --branch=v2.7.1 https://github.com/pytorch/pytorch.git && \
cd pytorch && \
git submodule sync && \
git submodule update --init --recursive && \
Expand All @@ -155,7 +155,7 @@ RUN \
export USE_MKLDNN=ON && \
export USE_QNNPACK=OFF && \
export USE_PYTORCH_QNNPACK=OFF && \
export PYTORCH_BUILD_VERSION=2.5.1 && \
export PYTORCH_BUILD_VERSION=2.7.1 && \
export PYTORCH_BUILD_NUMBER=1 && \
/usr/local/bin/python3.10 setup.py install && \
mkdir /usr/local/gcc103/include/pytorch && \
Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_aarch64_native_tester/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-native-build:15
FROM docker.elastic.co/ml-dev/ml-linux-aarch64-native-build:15a

MAINTAINER David Roberts <dave.roberts@elastic.co>

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_builder/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-linux-build:32
FROM docker.elastic.co/ml-dev/ml-linux-build:32a

MAINTAINER David Roberts <dave.roberts@elastic.co>

Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_dependency_builder_image/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-linux-build:32 AS builder
FROM docker.elastic.co/ml-dev/ml-linux-build:32a AS builder

# This is basically automating the setup instructions in build-setup/linux.md

Expand Down
6 changes: 3 additions & 3 deletions dev-tools/docker/linux_image/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ ARG build_dir=/usr/src
# Build gcc 10.3
RUN \
cd ${build_dir} && \
wget --quiet -O - http://ftpmirror.gnu.org/gcc/gcc-10.3.0/gcc-10.3.0.tar.gz | tar zxf - && \
wget --quiet -O - https://mirrorservice.org/sites/sourceware.org/pub/gcc/releases/gcc-10.3.0/gcc-10.3.0.tar.gz | tar zxf - && \
cd gcc-10.3.0 && \
contrib/download_prerequisites && \
sed -i -e 's/$(SHLIB_LDFLAGS)/-Wl,-z,relro -Wl,-z,now $(SHLIB_LDFLAGS)/' libgcc/config/t-slibgcc && \
Expand Down Expand Up @@ -154,7 +154,7 @@ gpgkey=https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.
# If the PyTorch branch is changed also update PYTORCH_BUILD_VERSION
RUN \
cd ${build_dir} && \
git -c advice.detachedHead=false clone --depth=1 --branch=v2.5.1 https://github.com/pytorch/pytorch.git && \
git -c advice.detachedHead=false clone --depth=1 --branch=v2.7.1 https://github.com/pytorch/pytorch.git && \
cd pytorch && \
git submodule sync && \
git submodule update --init --recursive && \
Expand All @@ -169,7 +169,7 @@ RUN \
export USE_QNNPACK=OFF && \
export USE_PYTORCH_QNNPACK=OFF && \
export USE_XNNPACK=OFF && \
export PYTORCH_BUILD_VERSION=2.5.1 && \
export PYTORCH_BUILD_VERSION=2.7.1 && \
export PYTORCH_BUILD_NUMBER=1 && \
export MAX_JOBS=10 && \
/usr/local/bin/python3.10 setup.py install && \
Expand Down
2 changes: 1 addition & 1 deletion dev-tools/docker/linux_tester/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#

# Increment the version here when a new tools/3rd party components image is built
FROM docker.elastic.co/ml-dev/ml-linux-build:32
FROM docker.elastic.co/ml-dev/ml-linux-build:32a

MAINTAINER David Roberts <dave.roberts@elastic.co>

Expand Down
4 changes: 2 additions & 2 deletions dev-tools/download_macos_deps.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ case `uname -m` in
ARCHIVE=local-x86_64-apple-macosx12.0-1.tar.bz2
;;
arm64)
ARCHIVE=local-arm64-apple-macosx11.1-12.tar.bz2
ARCHIVE=local-arm64-apple-macosx12.2-1.tar.bz2
;;

*)
Expand All @@ -38,7 +38,7 @@ case `uname -m` in

esac

URL="https://s3-eu-west-2.amazonaws.com/ml-cpp-artifacts/dependencies/$ARCHIVE"
URL="https://ml-cpp-artifacts.s3.eu-west-2.amazonaws.com/dependencies/$ARCHIVE"

echo "Downloading dependencies from $URL"
cd "$TMPDIR" && curl -s -S --retry 5 -O "$URL"
Expand Down
6 changes: 4 additions & 2 deletions dev-tools/download_windows_deps.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,13 @@
# limitation.
#
$ErrorActionPreference="Stop"
# TODO: Fix the windows build and use the latest archive
# $Archive="usr-x86_64-windows-2016-16.zip"
$Archive="usr-x86_64-windows-2016-15.zip"
$Destination="C:\"
# If PyTorch is not version 2.5.1 then we need the latest download
# If PyTorch is not version 2.7.1 then we need the latest download
if (!(Test-Path "$Destination\usr\local\include\pytorch\torch\csrc\api\include\torch\version.h") -Or
!(Select-String -Path "$Destination\usr\local\include\pytorch\torch\csrc\api\include\torch\version.h" -Pattern "2.5.1" -Quiet)) {
!(Select-String -Path "$Destination\usr\local\include\pytorch\torch\csrc\api\include\torch\version.h" -Pattern "2.7.1" -Quiet)) {
Remove-Item "$Destination\usr" -Recurse -Force -ErrorAction Ignore
$ZipSource="https://storage.googleapis.com/elastic-ml-public/dependencies/$Archive"
$ZipDestination="$Env:TEMP\$Archive"
Expand Down
6 changes: 6 additions & 0 deletions docs/CHANGELOG.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@

//=== Regressions

== {es} version 8.19.8

=== Enhancements

* Update the PyTorch library to version 2.7.1. (See {ml-pull}2882[#2882].)

== {es} version 8.19.0

=== Enhancements
Expand Down