Merge branch '2.10.pr' into 'trunk'

2.10.pr See merge request wdc/compss/framework!124
bsc-wdc · Nov 12, 2021 · db2a499 · db2a499
2 parents ae18cdf + 9642de3
commit db2a499
Show file tree

Hide file tree

Showing 24 changed files with 699 additions and 83 deletions.
diff --git a/RELEASE_NOTES b/RELEASE_NOTES
@@ -6,37 +6,32 @@ COMP Superscalar (COMPSs) is a programming model that aims to ease the developme
 such as Clusters, Grids and Clouds. COMP Superscalar also features a runtime system that exploits the inherent parallelism of 
 applications at execution time.
 
-Release number: 2.9 (Jasmine) 
-Release date: June-2021
+Release number: 2.10 (Kumquat) 
+Release date: November-2021
 -------------------------------
 New features
          * Runtime:
-		- Support for nested tasks in agents deployment 
-		- New application time out functionality to enable a controlled finalization of applications before the wall_clock_limit 
-		- New flags to simplify application debugging (--keep_workingdir, --gen_coredump)
-		- Support for loading the application environment from scripts
-		- Support for tracing in agents environment.
-		- Partial support for OSX systems. 
+		- Support for http requests (@http decorator). 
+		- Support for Java versions higher than 8.
+		- Enable grouping processes in MPI tasks with the processes_per_node flag. 
+		- Partial support for OSX systems 
 
          * Python:
-		- Support for Optional parameters and default values.
-		- Support for monitoring task status from Jupyter notebooks.
-		- Memory profile enabled
-		- Support for Python workers cache.
-		- Support for dynamic number of returns override at task invocation.
+		- PyArrow object serialization support
+		- Cache profiling enabled
 
-	* DDS:
-		- New methods and optimizations in DDS class.
 Improvements:
-        - Enabling the pass of extra flags for the queue system flags from enqueue_compss.
-        - Support for multiple data layout in MPI tasks
-	- Improvements in tracing system. More events and cfg
-	- Enabling a flag to change extrae configuration file for python processes.
-	- Configuration files for Barbora system.
+	- Fixes in managing directory parameters.
+        - Improvements in asynchronous file system operations.
+	- Improvements visualizing collections in task dependency graphs. 
+        - Improvements and fixes in the support for different MPI versions (COMPSS_MPIRUN_TYPE)
+	- Improvements in tracing system. Fixes in events and cfgs
+	- Configuration files for Karolina, Mahti and CTE-AMD system.
 	- Several Bug fixes.
 
 Known Limitations:
-	- OSX support is limited to Java and Python2 without CPU affinity (require to execute with --cpu_affinity=disable)
+	- Collections are not supported in http tasks
+	- OSX support is limited to Java and Python without CPU affinity (require to execute with --cpu_affinity=disable). We have also detected issues when several python3 versions are installed in the system. Tracing is not available.
         - Reduce operations can consume more disk space than the manually programmed n-ary reduction
 	- Objects used as task parameters must be serializable.
         - Tasks that invoke Numpy and MKL may experience issues if a different MKL threads count is used in different tasks. This is due to the fact that MKL reuses threads in the different calls and it does not change the number of threads from one call to another. This can be also happen with other libraries implemented with OpenMP.

diff --git a/builders/buildscs b/builders/buildscs
@@ -12,9 +12,11 @@
   SCtarget=/apps/COMPSs/${BranchName^} # capitalize branch name
 
   # Install on each enabled supercomputer
-  ENABLED_SCS=("mn" "mt" "nord" "p9")
-  SCS_SCRIPT_NAMES=("buildMN" "buildMT" "buildNord" "buildP9")
+  #ENABLED_SCS=("mn" "mt" "nord" "p9" "cte-amd")
+  #SCS_SCRIPT_NAMES=("buildMN" "buildMT" "buildNord" "buildP9" "buildCTE-AMD")
 
+  ENABLED_SCS=("mn" "mt" "p9" "cte-amd")
+  SCS_SCRIPT_NAMES=("buildMN" "buildMT" "buildP9" "buildCTE-AMD")
   echo "Installing COMPSs branch ${BranchName} at SCs"
 
   for i in "${!ENABLED_SCS[@]}"; do

diff --git a/builders/scs/p9/buildP9_aux b/builders/scs/p9/buildP9_aux
@@ -44,14 +44,13 @@
   #######
 
   # DEFINE MODULES
-  modules="gcc openmpi boost java atlas/3.10.3 python/2.7.14"
+  modules="gcc/8.3.0 openmpi/4.0.1 boost java atlas/3.10.3 python/2.7.14"
   unload_before_python="ignore"
   py2_modules="python/2.7.14"
   run_before_py3_modules="ignore"
   py3_modules="python/3.6.5"
-  run_before_py3_other_modules="ignore"
-  py3_other_modules="ignore"
-  #py3_other_modules="Python/3.7.0-foss-2018b"
+  run_before_py3_other_modules="module load cuda/10.2 cudnn/7.6.4 nccl/2.4.8 tensorrt/6.0.1 openmpi/4.0.1 scalapack/2.0.2 fftw/3.3.8 szip/2.1.1 ffmpeg/4.2.1 opencv/4.1.1"
+  py3_other_modules="python/3.7.4_ML"
 
   # DEFINE CONFIGURATION PARAMETERS
   #(Environment vars must be scaped to get the correct value if not it could take a value previous to load the modules)

diff --git a/changelog b/changelog
@@ -473,7 +473,43 @@ Known Limitations:
         - Delete file calls for files used as input can produce a significant synchronization of the main code.
         - Defining a parameter as OUT is only allowed for files and collections of objects with a default constructor.
 
-For further information, please refer to “COMPSs User Manual: Application development guide”.
+Release number: 2.10 (Kumquat) 
+Release date: November-2021
+-------------------------------
+New features
+         * Runtime:
+		- Support for http requests (@http decorator). 
+		- Support for Java versions higher than 8.
+		- Enable grouping processes in MPI tasks with the processes_per_node flag. 
+		- Partial support for OSX systems 
+
+         * Python:
+		- PyArrow object serialization support
+		- Cache profiling enabled
+
+Improvements:
+	- Fixes in managing directory parameters.
+        - Improvements in asynchronous file system operations.
+	- Improvements visualizing collections in task dependency graphs. 
+        - Improvements and fixes in the support for different MPI versions (COMPSS_MPIRUN_TYPE)
+	- Improvements in tracing system. Fixes in events and cfgs
+	- Configuration files for Karolina, Mahti and CTE-AMD system.
+	- Several Bug fixes.
+
+Known Limitations:
+	- Collections are not supported in http tasks
+	- OSX support is limited to Java and Python without CPU affinity (require to execute with --cpu_affinity=disable). We have also detected issues when several python3 versions are installed in the system. Tracing is not available.
+        - Reduce operations can consume more disk space than the manually programmed n-ary reduction
+	- Objects used as task parameters must be serializable.
+        - Tasks that invoke Numpy and MKL may experience issues if a different MKL threads count is used in different tasks. This is due to the fact that MKL reuses threads in the different calls and it does not change the number of threads from one call to another. This can be also happen with other libraries implemented with OpenMP.
+        - C++ Objects declared as arguments in coarse-grain tasks must be passed as object pointers in order to have proper dependency management.
+        - Master as worker feature is not working for executions with persistent worker in C++.
+	- Coherence and concurrent writing in parameters annotated with the "Concurrent" direction must be managed by the underlying distributed storage system.
+	- Delete file calls for files used as input can produce a significant synchronization of the main code.
+	- Defining a parameter as OUT is only allowed for files and collections of objects with a default constructor.   
+
+For further information, please refer to the COMPSs Documentation at: 
+	https://compss-doc.readthedocs.io/en/stable/
 
 Please find more details about the COMP Superscalar framework at: 
-		http://compss.bsc.es/
+	http://compss.bsc.es/
diff --git a/compss/programming_model/bindings/python/src/pycompss/interactive.py b/compss/programming_model/bindings/python/src/pycompss/interactive.py
@@ -83,7 +83,7 @@
 STREAMING = False
 LOG_PATH = tempfile.mkdtemp()
 GRAPHING = False
-LINE_SEPARATOR = "******************************************************"
+LINE_SEPARATOR = "********************************************************"
 
 
 # Initialize multiprocessing
@@ -393,7 +393,7 @@ def start(log_level="off",                     # type: str
     # RUNTIME START
     ##############################################################
 
-    print("* - Starting COMPSs runtime...                       *")
+    print("* - Starting COMPSs runtime...                         *")
     sys.stdout.flush()  # Force flush
     compss_start(log_level, all_vars["trace"], True)
 
@@ -434,7 +434,7 @@ def start(log_level="off",                     # type: str
 
     # MAIN EXECUTION
     # let the user write an interactive application
-    print("* - PyCOMPSs Runtime started... Have fun!            *")
+    print("* - PyCOMPSs Runtime started... Have fun!              *")
     print(LINE_SEPARATOR)
 
     # Emit the application start event (the 0 is in the stop function)
@@ -447,26 +447,26 @@ def __show_flower__():
 
     :return: None
     """
-    print(LINE_SEPARATOR)                                            # NOSONAR # noqa
-    print("*************** PyCOMPSs Interactive *****************")  # NOSONAR # noqa
-    print(LINE_SEPARATOR)                                            # NOSONAR # noqa
-    print("*          .-~~-.--.           _____       _______   *")  # NOSONAR # noqa
-    print("*         :         )         |____ \     /  ___  \  *")  # NOSONAR # noqa
-    print("*   .~ ~ -.\       /.- ~~ .     ___) |    | (___) |  *")  # NOSONAR # noqa
-    print("*   >       `.   .'       <    / ___/     \____   /  *")  # NOSONAR # noqa
-    print("*  (         .- -.         )  | |___   _      /  /   *")  # NOSONAR # noqa
-    print("*   `- -.-~  `- -'  ~-.- -'   |_____| |_|    /__/    *")  # NOSONAR # noqa
-    print("*     (        :        )           _ _ .-:          *")  # NOSONAR # noqa
-    print("*      ~--.    :    .--~        .-~  .-~  }          *")  # NOSONAR # noqa
-    print("*          ~-.-^-.-~ \_      .~  .-~   .~            *")  # NOSONAR # noqa
-    print("*                   \ \ '     \ '_ _ -~              *")  # NOSONAR # noqa
-    print("*                    \`.\`.    //                    *")  # NOSONAR # noqa
-    print("*           . - ~ ~-.__\`.\`-.//                     *")  # NOSONAR # noqa
-    print("*       .-~   . - ~  }~ ~ ~-.~-.                     *")  # NOSONAR # noqa
-    print("*     .' .-~      .-~       :/~-.~-./:               *")  # NOSONAR # noqa
-    print("*    /_~_ _ . - ~                 ~-.~-._            *")  # NOSONAR # noqa
-    print("*                                     ~-.<           *")  # NOSONAR # noqa
-    print(LINE_SEPARATOR)                                            # NOSONAR # noqa
+    print(LINE_SEPARATOR)                                              # NOSONAR # noqa
+    print("**************** PyCOMPSs Interactive ******************")  # NOSONAR # noqa
+    print(LINE_SEPARATOR)                                              # NOSONAR # noqa
+    print("*          .-~~-.--.           _____      __   ______  *")  # NOSONAR # noqa
+    print("*         :         )         |____ \    /  | /  __  \ *")  # NOSONAR # noqa
+    print("*   .~ ~ -.\       /.- ~~ .     ___) |  /_  | | |  | | *")  # NOSONAR # noqa
+    print("*   >       `.   .'       <    / ___/     | | | |  | | *")  # NOSONAR # noqa
+    print("*  (         .- -.         )  | |___   _  | | | |__| | *")  # NOSONAR # noqa
+    print("*   `- -.-~  `- -'  ~-.- -'   |_____| |_| |_| \______/ *")  # NOSONAR # noqa
+    print("*     (        :        )           _ _ .-:            *")  # NOSONAR # noqa
+    print("*      ~--.    :    .--~        .-~  .-~  }            *")  # NOSONAR # noqa
+    print("*          ~-.-^-.-~ \_      .~  .-~   .~              *")  # NOSONAR # noqa
+    print("*                   \ \ '     \ '_ _ -~                *")  # NOSONAR # noqa
+    print("*                    \`.\`.    //                      *")  # NOSONAR # noqa
+    print("*           . - ~ ~-.__\`.\`-.//                       *")  # NOSONAR # noqa
+    print("*       .-~   . - ~  }~ ~ ~-.~-.                       *")  # NOSONAR # noqa
+    print("*     .' .-~      .-~       :/~-.~-./:                 *")  # NOSONAR # noqa
+    print("*    /_~_ _ . - ~                 ~-.~-._              *")  # NOSONAR # noqa
+    print("*                                     ~-.<             *")  # NOSONAR # noqa
+    print(LINE_SEPARATOR)                                              # NOSONAR # noqa
 
 
 def __print_setup__(verbose, all_vars):

diff --git a/...ss/runtime/adaptors/engine/src/main/java/es/bsc/compss/types/data/accessid/RAccessId.java b/...ss/runtime/adaptors/engine/src/main/java/es/bsc/compss/types/data/accessid/RAccessId.java
@@ -93,7 +93,7 @@ public int getRVersionId() {
      * @return {@code true} if the source data must be preserved, {@code false} otherwise.
      */
     public boolean isPreserveSourceData() {
-        return this.readDataVersion.hasMoreReaders();
+        return this.readDataVersion.hasMoreReaders() || this.readDataVersion.getDataInstanceId().getVersionId() == 1;
     }
 
     @Override

diff --git a/...s/runtime/resources/commons/src/main/java/es/bsc/compss/connectors/AbstractConnector.java b/...s/runtime/resources/commons/src/main/java/es/bsc/compss/connectors/AbstractConnector.java
@@ -16,7 +16,6 @@
  */
 package es.bsc.compss.connectors;
 
-import es.bsc.compss.comm.Comm;
 import es.bsc.compss.connectors.utils.CreationThread;
 import es.bsc.compss.connectors.utils.DeadlineThread;
 import es.bsc.compss.connectors.utils.DeletionThread;
@@ -28,24 +27,15 @@
 import es.bsc.compss.types.resources.ShutdownListener;
 import es.bsc.compss.types.resources.description.CloudMethodResourceDescription;
 
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileNotFoundException;
-import java.io.IOException;
-import java.io.ObjectInputStream;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
 import java.util.TreeSet;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.Semaphore;
 
-import jdk.nashorn.internal.parser.JSONParser;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
-import org.json.JSONArray;
-import org.json.JSONObject;
-import org.json.JSONTokener;
 
 
 /**

diff --git a/compss/runtime/scripts/system/commons/version.sh b/compss/runtime/scripts/system/commons/version.sh
@@ -2,7 +2,7 @@
 # Displays version
 ###############################################
 show_version() {
-  echo "COMPSs version 2.9 Jasmine"
+  echo "COMPSs version 2.10 Kumquat"
   echo " "
 }
 
@@ -12,12 +12,12 @@ show_version() {
 show_full_version() {
   cat << EOF
                              
-        .-~~-.--.             _____      _______
-       :         )           |____ \\    /  ___  \\
- .~ ~ -.\\       /.- ~~ .      ___) |    | (___) |
- >       \`.   .'       <     / ___/      \\____  | 
-(         .- -.         )   | |____   _   ____) | 
- \`- -.-~  \`- -'  ~-.- -'    |______| |_| |______/
+        .-~~-.--.             _____       _   ______
+       :         )           |____ \\     / | /  __  \\
+ .~ ~ -.\\       /.- ~~ .      ___) |    /_ | | |  | |
+ >       \`.   .'       <     / ___/      | | | |  | |
+(         .- -.         )   | |____   _  | | | |__| | 
+ \`- -.-~  \`- -'  ~-.- -'    |______| |_| |_| \\______/
    (        :        )           _ _ .-:
     ~--.    :    .--~        .-~  .-~  }
         ~-.-^-.-~ \\       .~  .-~   .~
@@ -37,9 +37,11 @@ EOF
 ###############################################
 show_flower() {
   cat << EOF
-Jasmine is a genus of shrubs and vines in the olive family (Oleaceae). The flowers are typically around 2.5 cm (0.98 in) in diameter. They are white or yellow in color, although in rare instances they can be slightly reddish. The flowers are borne in cymose clusters with a minimum of three flowers, though they can also be solitary on the ends of branchlets. Each flower has about four to nine petals, two locules, and one to four ovules. They have two stamens with very short filaments. The bracts are linear or ovated. 
+Kumquats are a group of small fruit-bearing trees in the flowering plant family Rutaceae. They were previously classified as forming the now-historical genus Fortunella, or placed within Citrus, sensu lato.
+They are slow-growing evergreen shrubs or short trees that stand 2.5 to 4.5 meters (8 to 15 ft) tall, with dense branches, sometimes bearing small thorns. The leaves are dark glossy green, and the flowers are white, similar to other citrus flowers, and can be borne singly or clustered within the leaves' axils. Depending on size, the kumquat tree can produce hundreds or even thousands of fruits each year.
+
 Check more information:
-   https://en.wikipedia.org/wiki/Jasmine
+   https://en.wikipedia.org/wiki/Kumquat
 
 EOF
 }
@@ -50,20 +52,20 @@ EOF
 show_recipe() {
   cat << EOF
 
-Jasmine - Cocktail 
+Kumquat Quest - Cocktail 
 
-From: https://www.liquor.com/recipes/jasmine/
+From: https://cocktailsdistilled.com/recipe/kumquat-quest/
 
 Ingredients:
 
-    1 1/2 oz Gin
-    1/4 oz Campari
-    1/4 oz Orange liqueur
-    3/4 oz Fresh lemon juice
-    Garnish: Lemon twist
+    2 oz Vodka
+    1/2 oz Grand Marnier
+    1/2 oz Lime Juice
+    Sugar, Demerara Brown
+    3-4 Kumquats
 
 Instructions:
-    Place all ingredients in iced cocktail shaker strain. Shake and add sugar to taste. Strain into a chilled cocktail glass. Garnish with a lemon twist
+    Cut all the kumquats in half and put all the rest of the ingredients into a highball glass. Muddle all and add crushed ice. Float Grand Marnier.
 
 EOF
 }
diff --git a/pom.xml b/pom.xml
@@ -130,7 +130,7 @@
         <repository>
             <id>compss-repo</id>
             <name>BSC Workflows and Distributed Computing group</name>
-            <url>http://compss.bsc.es/artifactory/libs-release</url>
+            <url>https://compss.bsc.es/artifactory/libs-release</url>
         </repository>
     </repositories>