Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Yan 919 - Workflow signature mechanism #133

Merged
merged 372 commits into from
May 12, 2022
Merged

Yan 919 - Workflow signature mechanism #133

merged 372 commits into from
May 12, 2022
This pull request is big! We’re only showing the most recent 250 commits.

Commits on Jul 27, 2020

  1. Merge from original repo

    pritchardn committed Jul 27, 2020
    Configuration menu
    Copy the full SHA
    34548d3 View commit details
    Browse the repository at this point in the history

  2. Importing typing.re

    pritchardn committed Jul 27, 2020
    Configuration menu
    Copy the full SHA
    fe14f99 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2020

  1. Merge branch 'accumulate_repeat' 

    # Conflicts:
#	daliuge-common/dlg/common/reproducibility/constants.py
#	daliuge-runtime/dlg/graph_loader.py
    pritchardn committed Jul 28, 2020
    Configuration menu
    Copy the full SHA
    2f7e3c0 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/repro_forwarding'

    pritchardn committed Jul 28, 2020
    Configuration menu
    Copy the full SHA
    76e0edb View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2020

  1. Adds leaf agglomeration function

    pritchardn committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    3b3e1fb View commit details
    Browse the repository at this point in the history

  2. Renames 'leaves' to 'signature' indicating the finality of the collec… 

    …ted leaf hashes.
    pritchardn committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    f9f223b View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2020

  1. Adds 'Reproduce' constant

    pritchardn committed Aug 3, 2020
    Configuration menu
    Copy the full SHA
    1274154 View commit details
    Browse the repository at this point in the history

  2. Adds Reproduce to accumulate_lgt_drop_dat 

    Adds other TODOs critical for Repetition that were missed before.
    pritchardn committed Aug 3, 2020
    Configuration menu
    Copy the full SHA
    b82f5bd View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2020

  1. Somehow 'reproduciblity' snuck back in again...

    pritchardn committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    76170a3 View commit details
    Browse the repository at this point in the history

  2. Changes EXPERIMENTAL to REPEAT

    pritchardn committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    8c1bc3f View commit details
    Browse the repository at this point in the history

  3. Adds support for new DALiuGE Drop Categories. Adds support for NULL a… 

    …nd JSON drops (boilerplate) in Repeat accumulate
    pritchardn committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    4881d17 View commit details
    Browse the repository at this point in the history

  4. Adds condition to accumulate_pgt_unroll_drop_data for varying reprodu… 

    …cibility level support.
    pritchardn committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    346d10f View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into reproduce 

    # Conflicts:
#	daliuge-common/dlg/common/reproducibility/constants.py
#	daliuge-common/dlg/common/reproducibility/reproducibility.py
    pritchardn committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    4b9e52f View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2020

  1. Adds reproducibility behaviour for data-graph-minor blockdag generation

    pritchardn committed Aug 10, 2020
    Configuration menu
    Copy the full SHA
    f5ed210 View commit details
    Browse the repository at this point in the history

  2. Changes implementation of blockdag generation on the data-graph-minor 

    Instead we remove duplicate values at the block-building stage. Also fixes a pgt/pg blockdag parenthash appending bug
    pritchardn committed Aug 10, 2020
    Configuration menu
    Copy the full SHA
    68f30a5 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2020

  1. Adds python modules (now in the DALiuGE codebase, I cannot see anothe… 

    …r way forward without major time).
    pritchardn committed Aug 11, 2020
    Configuration menu
    Copy the full SHA
    2f9f550 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2020

  1. Adds basic DROP-level reproducibility functions.

    pritchardn committed Aug 13, 2020
    Configuration menu
    Copy the full SHA
    3dc5fcf View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2020

  1. Fixes many broken edge cases with building a data-graph-minor blockDAG

    pritchardn committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    0158180 View commit details
    Browse the repository at this point in the history

  2. Adds reproduce behaviour (simple, basic) for FileDROPs and MemoryDROPs

    pritchardn committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    0ba29e6 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2020

  1. A very basic NGAS implementation

    pritchardn committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    cd0ed44 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2020

  1. Adds Reproducibility data for RDBMS drops

    pritchardn committed Aug 27, 2020
    Configuration menu
    Copy the full SHA
    8d13e67 View commit details
    Browse the repository at this point in the history

  2. Updates TODO

    pritchardn committed Aug 27, 2020
    Configuration menu
    Copy the full SHA
    24f3b08 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2020

  1. Converts all runtime drop_data structures to dictionaries (were previ… 

    …ously lists)
    pritchardn committed Sep 2, 2020
    Configuration menu
    Copy the full SHA
    e07be82 View commit details
    Browse the repository at this point in the history

  2. Drop runtime data is now appended to the drop structure directly (eff… 

    …ectively initializing the runtime drop data)
    pritchardn committed Sep 2, 2020
    Configuration menu
    Copy the full SHA
    c01f21f View commit details
    Browse the repository at this point in the history

  3. separates Runtime data from PG data. It's more accurate vs. conflatin… 

    …g the two.
    @pritchardn
    pritchardn authored and pritchardn committed Sep 2, 2020
    Configuration menu
    Copy the full SHA
    6d1109d View commit details
    Browse the repository at this point in the history

  4. Merge branch 'ExperimentSetup'

    pritchardn committed Sep 2, 2020
    Configuration menu
    Copy the full SHA
    f440640 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2020

  1. rmode_supported now supports replication

    pritchardn committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    fa3b8a1 View commit details
    Browse the repository at this point in the history

  2. Drops now implement replication methods.

    pritchardn committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    912ebcb View commit details
    Browse the repository at this point in the history

  3. Drops now actually accumulate replicate information

    pritchardn committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    7164c72 View commit details
    Browse the repository at this point in the history

  4. Translator reproducibility functions now accumulate replication data

    pritchardn committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    601a7c9 View commit details
    Browse the repository at this point in the history

  5. Fixes typo in documentation

    pritchardn committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    470ba0e View commit details
    Browse the repository at this point in the history

  6. BlockDAG construction now adds both the full-graph and data-driven mi… 

    …nor, completing the implementation of Replication.
    pritchardn committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    4503858 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2020

  1. Adds some error checking

    pritchardn committed Sep 15, 2020
    Configuration menu
    Copy the full SHA
    0693017 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2020

  1. Minor spacing fix

    pritchardn committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    c344480 View commit details
    Browse the repository at this point in the history

  2. Adds rflag_caster and docstrings to constants.py

    pritchardn committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    bd231cc View commit details
    Browse the repository at this point in the history

  3. Adds use of rflag_caster

    pritchardn committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    8536f88 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2020

  1. Adds simple numpy average and my average function apps.

    @pritchardn
    pritchardn committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    e89bcbb View commit details
    Browse the repository at this point in the history

  2. Adjustment made to pgt code to use 'dt' instead of 'app' to determine… 

    … the type of each drop.
    @pritchardn
    pritchardn committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    5a24032 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2020

  1. Dictionaries inserted are inserted into MerkleTrees as .items()

    @pritchardn
    pritchardn committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    395229e View commit details
    Browse the repository at this point in the history

  2. Fixes example apps

    @pritchardn
    pritchardn committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    5854df5 View commit details
    Browse the repository at this point in the history

  3. Reproducibility now only tracks data-drops at the DAG-frontier. 

    Data-Drops with no incoming or no outgoing edges.
    @pritchardn
    pritchardn committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    f3c0f1c View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2021

  1. Adds high-level recompute and total-replication support

    @pritchardn
    pritchardn committed Jan 7, 2021
    Configuration menu
    Copy the full SHA
    d4dd2fb View commit details
    Browse the repository at this point in the history

  2. Adds base recompute and total replication data code.

    @pritchardn
    pritchardn committed Jan 7, 2021
    Configuration menu
    Copy the full SHA
    6ce8512 View commit details
    Browse the repository at this point in the history

Commits on Jan 8, 2021

  1. Adds low-level recompute information for bash_shell_app.py

    @pritchardn
    pritchardn committed Jan 8, 2021
    Configuration menu
    Copy the full SHA
    fa95c7a View commit details
    Browse the repository at this point in the history

  2. Adds low-level recompute information for dockerapp.py

    @pritchardn
    pritchardn committed Jan 8, 2021
    Configuration menu
    Copy the full SHA
    5d483c3 View commit details
    Browse the repository at this point in the history

  3. Adds low-level recompute information for pyfunc.py

    @pritchardn
    pritchardn committed Jan 8, 2021
    Configuration menu
    Copy the full SHA
    6dcd022 View commit details
    Browse the repository at this point in the history

  4. Adds low-level recompute information for mpi.py

    @pritchardn
    pritchardn committed Jan 8, 2021
    Configuration menu
    Copy the full SHA
    04a9b9c View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2021

  1. Adds recompute functionality to dynlib.py 

    Stores a dictionary of CDlgApp variables cast to string representations.
    @pritchardn
    pritchardn committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    b732afe View commit details
    Browse the repository at this point in the history

  2. Moves generate_recompute_data in dynlib.py to the Drop-inheriting cla… 

    …sses. Adds 'status' field
    @pritchardn
    pritchardn committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    364e694 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2021

  1. Exchanges PGS/PGT behaviour for repeat/total_replication and recomput… 

    …e/computational_replication
    @pritchardn
    pritchardn committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    78b3810 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2021

  1. Adds a signal generator drop

    @pritchardn
    pritchardn committed Jan 18, 2021
    Configuration menu
    Copy the full SHA
    0a7692e View commit details
    Browse the repository at this point in the history

  2. Adds a window generator drop and refines signal generator

    @pritchardn
    pritchardn committed Jan 18, 2021
    Configuration menu
    Copy the full SHA
    172a85c View commit details
    Browse the repository at this point in the history

  3. Separates Lowpass example specific apps into apps_lowpass.py

    @pritchardn
    pritchardn committed Jan 18, 2021
    Configuration menu
    Copy the full SHA
    e40ab02 View commit details
    Browse the repository at this point in the history

  4. Adds add_noise drop to Lowpass example.

    @pritchardn
    pritchardn committed Jan 18, 2021
    Configuration menu
    Copy the full SHA
    6323573 View commit details
    Browse the repository at this point in the history

  5. Adds pointwise filter to apps_lowpass.py

    @pritchardn
    pritchardn committed Jan 18, 2021
    Configuration menu
    Copy the full SHA
    39a1f5f View commit details
    Browse the repository at this point in the history

Commits on Jan 20, 2021

  1. Adds full support for lowpass filter implementations

    @pritchardn
    pritchardn committed Jan 20, 2021
    Configuration menu
    Copy the full SHA
    974e780 View commit details
    Browse the repository at this point in the history

  2. Adds ID to parenthash dictionaries (not lists). Resolves (un)intentio… 

    …nal duplicates.
    @pritchardn
    pritchardn committed Jan 20, 2021
    Configuration menu
    Copy the full SHA
    1f59fbe View commit details
    Browse the repository at this point in the history

  3. Block hashes are made only from the parent hash values.

    @pritchardn
    pritchardn committed Jan 20, 2021
    Configuration menu
    Copy the full SHA
    c0155d1 View commit details
    Browse the repository at this point in the history

  4. Bug fixes cuda fft

    @pritchardn
    pritchardn committed Jan 20, 2021
    Configuration menu
    Copy the full SHA
    b768333 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2021

  1. Changes drop hash to common_hash function

    @pritchardn
    pritchardn committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    c4c7ba0 View commit details
    Browse the repository at this point in the history

  2. Updates original drophash testing functions

    @pritchardn
    pritchardn committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    6d0fa40 View commit details
    Browse the repository at this point in the history

  3. Changing a drop's flag will now automatically recommit it's reproduci… 

    …bilty data if it was previously committed.
    @pritchardn
    pritchardn committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    afa3034 View commit details
    Browse the repository at this point in the history

  4. Adds reproducibility tests for bashShellApps

    @pritchardn
    pritchardn committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    79d08e7 View commit details
    Browse the repository at this point in the history

  5. Adds reproducibility tests for AbstractDrops

    @pritchardn
    pritchardn committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    adfc2e4 View commit details
    Browse the repository at this point in the history

  6. Adds reproducibility tests for PyFuncApps

    @pritchardn
    pritchardn committed Jan 22, 2021
    Configuration menu
    Copy the full SHA
    417e36d View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2021

  1. Drops no longer store 'rank' at the PGT/PGS layer when rerunning.

    @pritchardn
    pritchardn committed Jan 25, 2021
    Configuration menu
    Copy the full SHA
    a19c482 View commit details
    Browse the repository at this point in the history

  2. Drops only store 'rank' when recomputing or replicating_computationally.

    @pritchardn
    pritchardn committed Jan 25, 2021
    Configuration menu
    Copy the full SHA
    4fa9531 View commit details
    Browse the repository at this point in the history

  3. Data drops store filename and dirname when recomputing or replicating… 

    … computationally
    @pritchardn
    pritchardn committed Jan 25, 2021
    Configuration menu
    Copy the full SHA
    2047bc3 View commit details
    Browse the repository at this point in the history

  4. Only data drops store any LGT,LG,PGS,PGT or PG data when reproducing.

    @pritchardn
    pritchardn committed Jan 25, 2021
    Configuration menu
    Copy the full SHA
    ff7a1b3 View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2021

  1. Separates writing out function.

    @pritchardn
    pritchardn committed Jan 28, 2021
    Configuration menu
    Copy the full SHA
    a33ecb8 View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2021

  1. Removes generate_reproduce_data for the processing components.

    @pritchardn
    pritchardn committed Jan 29, 2021
    Configuration menu
    Copy the full SHA
    f92226f View commit details
    Browse the repository at this point in the history

  2. Lowpass filter apps now write out their numpy arrays as byte-strings.… 

    … Now data is passed around correctly.
    @pritchardn
    pritchardn committed Jan 29, 2021
    Configuration menu
    Copy the full SHA
    42618a6 View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2021

  1. Merge remote-tracking branch 'upstream/master' 

    # Conflicts:
#	.travis.yml
#	daliuge-runtime/dlg/utils.py
#	daliuge-runtime/test/manager/test_dm.py
#	daliuge-translator/dlg/dropmake/web/lg_web.py
#	daliuge-translator/dlg/dropmake/web/pg_viewer.html
#	daliuge-translator/dlg/translator/tool_commands.py
#	daliuge-translator/test/dropmake/test_pg_gen.py
    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    5047ef1 View commit details
    Browse the repository at this point in the history

  2. Changes fill to dlg_fill

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    697afd4 View commit details
    Browse the repository at this point in the history

  3. rflag caster now catches None case

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    eca5233 View commit details
    Browse the repository at this point in the history

  4. deploy tests pass with added reproducibility considerations.

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    4087164 View commit details
    Browse the repository at this point in the history

  5. Graph loader now tests for a None dropSpecList upon loading

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    a78b27c View commit details
    Browse the repository at this point in the history

  6. Composite manager now appends initial reprodata to all partitions.

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    1a89fd5 View commit details
    Browse the repository at this point in the history

  7. lg_buildblockdag and build_blockdag now return leaves and visited nodes. 

    This change cascades into downstream methods
    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    19713c6 View commit details
    Browse the repository at this point in the history

  8. test_dm now includes reprodata considerations

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    4dac407 View commit details
    Browse the repository at this point in the history

  9. minor bug-fix in test_drophash

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    73a7eee View commit details
    Browse the repository at this point in the history

  10. test_rest now had reprodata considerations

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    1932f99 View commit details
    Browse the repository at this point in the history

  11. test_session now had reprodata considerations

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    7a4cbe1 View commit details
    Browse the repository at this point in the history

  12. test_toposort now runs graphs through reprodata initialization and us… 

    …es the returned 'visited' list
    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    b802c9e View commit details
    Browse the repository at this point in the history

  13. The mother of all merges from daliuge/ray-test

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    f84dd1c View commit details
    Browse the repository at this point in the history

  14. FileDROP setComplete now fires the appropriate reproducibility event.

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    761bf06 View commit details
    Browse the repository at this point in the history

  15. Adjusts import statements for ray update

    @pritchardn
    pritchardn committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    72fb888 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2021

  1. Very minor adjustment to reproducibility dag building behaviour. 

    The Graph-minor traversal method is only used when Reproducing specifically.
    @pritchardn
    pritchardn committed Mar 11, 2021
    Configuration menu
    Copy the full SHA
    dce6f39 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2021

  1. Adds rerun thesis graph tests

    @pritchardn
    pritchardn committed Apr 15, 2021
    Configuration menu
    Copy the full SHA
    040618e View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2021

  1. Adds system summary to lowpass apps

    @pritchardn
    pritchardn committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    f43a33e View commit details
    Browse the repository at this point in the history

  2. Updates results with system information

    @pritchardn
    pritchardn committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    562b852 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2021

  1. Performs the merge

    @pritchardn
    pritchardn committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    bbb1457 View commit details
    Browse the repository at this point in the history

  2. Adds merklelib dependency for reproducibility

    @pritchardn
    pritchardn committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    4abab3b View commit details
    Browse the repository at this point in the history

  3. Changes ''' to """ for consistency in plasma.py

    @pritchardn
    pritchardn committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    6feff41 View commit details
    Browse the repository at this point in the history

  4. Adds a first cut at a reproducibility implementation for plasma writers.

    @pritchardn
    pritchardn committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    b97b499 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2021

  1. NGASDrop now uses the actual Ngas FileID 

    NGASIO now paginates data-reads (although actual reading happens once)
NGASIO now _writes with bytes and strings
_close updates method to _post method.
    @pritchardn
    pritchardn committed May 24, 2021
    Configuration menu
    Copy the full SHA
    bbf52bf View commit details
    Browse the repository at this point in the history

  2. Removes duplicate mount points in dockerapp.py by using list->set->li… 

    …st trick.
    @pritchardn
    pritchardn committed May 24, 2021
    Configuration menu
    Copy the full SHA
    ee9c520 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2021

  1. Adds a quick and dirty DirectoryIO class in io.py to recursively read… 

    … a directory's file contents into a buffer
    @pritchardn
    pritchardn committed Jun 22, 2021
    Configuration menu
    Copy the full SHA
    234836b View commit details
    Browse the repository at this point in the history

  2. If a file is in fact a directory, use a DirectoryIO object

    @pritchardn
    pritchardn committed Jun 22, 2021
    Configuration menu
    Copy the full SHA
    fd3e4c9 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2021

  1. Fixes faulty treatment of group nodes in reproducibility.py

    @pritchardn
    pritchardn committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    fab3ba7 View commit details
    Browse the repository at this point in the history

  2. convert_contruct now appends LGT group reprodata to input application… 

    …s in dm_utils.py
    @pritchardn
    pritchardn committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    bec5f23 View commit details
    Browse the repository at this point in the history

  3. building the blockdag when rerunning now only includes nodes from the… 

    … original logical graph
    @pritchardn
    pritchardn committed Jun 28, 2021
    Configuration menu
    Copy the full SHA
    850a42a View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2021

  1. Adds icrar accelerate cli leap_drops into daliuge for simpler calling.

    @pritchardn
    pritchardn committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    aea3fa3 View commit details
    Browse the repository at this point in the history

  2. Adds conditional checking for scatter and gather fields existing. 

    Consider adding better support for fields later on.
    @pritchardn
    pritchardn committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    5a7e385 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2021

  1. Moves leap drops from daliuge fork to windlass

    @pritchardn
    pritchardn committed Jul 19, 2021
    Configuration menu
    Copy the full SHA
    86c3bec View commit details
    Browse the repository at this point in the history

  2. Style adjustments by pylint for reproducibility.py

    @pritchardn
    pritchardn committed Jul 19, 2021
    Configuration menu
    Copy the full SHA
    edd3086 View commit details
    Browse the repository at this point in the history

  3. Style adjustments by pylint for apps.py

    @pritchardn
    pritchardn committed Jul 19, 2021
    Configuration menu
    Copy the full SHA
    53a0f0f View commit details
    Browse the repository at this point in the history

  4. Style adjustments by pylint for constants.py

    @pritchardn
    pritchardn committed Jul 19, 2021
    Configuration menu
    Copy the full SHA
    b0a4bb4 View commit details
    Browse the repository at this point in the history

  5. Style adjustments by pylint for apps_lowpass.py

    @pritchardn
    pritchardn committed Jul 19, 2021
    Configuration menu
    Copy the full SHA
    28b9e61 View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2021

  1. Makes build_blockdag track and report the order of node visits, not j… 

    …ust the count.
    @pritchardn
    pritchardn committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    9826247 View commit details
    Browse the repository at this point in the history

  2. Adds toposorting tests for all graph abstraction levels.

    @pritchardn
    pritchardn committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    5817902 View commit details
    Browse the repository at this point in the history

  3. Adds Logical Blockdag Rerun tests

    @pritchardn
    pritchardn committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    03106b0 View commit details
    Browse the repository at this point in the history

  4. Adds behaviour for NOTHING rmode: Does not process parents at all.

    @pritchardn
    pritchardn committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    f71ab6b View commit details
    Browse the repository at this point in the history

  5. Adds logical blockdag testing for repeating, recomputing, reproducing… 

    …, replicating and nothing.
    @pritchardn
    pritchardn committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    04964e4 View commit details
    Browse the repository at this point in the history

  6. Adds licence to reproducibility.py

    @pritchardn
    pritchardn committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    c9ea9f5 View commit details
    Browse the repository at this point in the history

  7. Separates out blockdag construction testing into a separate module. 

    Renames variables in test_toposort.py for pylint perfection.
    @pritchardn
    pritchardn committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    22c4105 View commit details
    Browse the repository at this point in the history

  8. Typo fixes in docstrings

    @pritchardn
    pritchardn committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    8290cd1 View commit details
    Browse the repository at this point in the history

  9. Makes rerunning behaviour more explicit for correctness.

    @pritchardn
    pritchardn committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    0983526 View commit details
    Browse the repository at this point in the history

  10. Adds test_pg_blockdag.py which tests blockdag construction for all rm… 

    …ode standards.
    @pritchardn
    pritchardn committed Jul 21, 2021
    Configuration menu
    Copy the full SHA
    0404877 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2021

  1. Adds __init__.py to reproducibility testing folder

    @pritchardn
    pritchardn committed Jul 22, 2021
    Configuration menu
    Copy the full SHA
    995f528 View commit details
    Browse the repository at this point in the history

  2. Adds a full-length 'nothing' standard integration test.

    @pritchardn
    pritchardn committed Jul 22, 2021
    Configuration menu
    Copy the full SHA
    bf4f0b0 View commit details
    Browse the repository at this point in the history

  3. Adds a scattering graph and test to demonstrate how rerunning works.

    @pritchardn
    pritchardn committed Jul 22, 2021
    Configuration menu
    Copy the full SHA
    e484ef4 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2021

  1. Updates test graphs with latest EAGLE data.

    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    e3ee74d View commit details
    Browse the repository at this point in the history

  2. Adds licence, implements AccumulateLGTRerunData and AccumulateLGRerun… 

    …Data
    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    c411033 View commit details
    Browse the repository at this point in the history

  3. Implements AccumulatePGTUnrollRerunData test and cleans up some varia… 

    …bles.
    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    f343630 View commit details
    Browse the repository at this point in the history

  4. Implements AccumulatePGTPartitionRerunData and AccumulatePGPartitionR… 

    …erunData
    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    e44793f View commit details
    Browse the repository at this point in the history

  5. Implements Accumulate[X]PartitionRepeatData

    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    4c669dc View commit details
    Browse the repository at this point in the history

  6. Improves key comparison in all tests.

    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    631ceeb View commit details
    Browse the repository at this point in the history

  7. Implements accumulatedata Recompute tests

    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    06fe3fa View commit details
    Browse the repository at this point in the history

  8. Implements accumulatedata Reproduce tests

    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    888dca0 View commit details
    Browse the repository at this point in the history

  9. Implements accumulatedata Replicate-Sci tests

    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    e5f4a9e View commit details
    Browse the repository at this point in the history

  10. Tempfiles are now used correctly.

    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    785d6e0 View commit details
    Browse the repository at this point in the history

  11. Adds AccumulateXReplicateCompData tests and skips unimplemented tests

    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    6580531 View commit details
    Browse the repository at this point in the history

  12. Adds AccumulateXReplicateTotalData tests

    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    446b4e5 View commit details
    Browse the repository at this point in the history

  13. Fixes test_accumulatedata.py for pylint

    @pritchardn
    pritchardn committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    f867e49 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2021

  1. Adds integration tests for HelloWorld example - all rmodes down to Ph… 

    …ysical Graph.
    @pritchardn
    pritchardn committed Jul 28, 2021
    Configuration menu
    Copy the full SHA
    c68ca4d View commit details
    Browse the repository at this point in the history

  2. Style changes to reproducibility tests for pylint score.

    @pritchardn
    pritchardn committed Jul 28, 2021
    Configuration menu
    Copy the full SHA
    2e9dfaf View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2021

  1. Style changes to reproducibility tests for pylint score.

    @pritchardn
    pritchardn committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    c7aed57 View commit details
    Browse the repository at this point in the history

  2. Adds functionality for logical graph templates to have pre-set rmodes… 

    … on a per-drop basis.
    @pritchardn
    pritchardn committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    e2079a8 View commit details
    Browse the repository at this point in the history

  3. Tests a split-rmode graph.

    @pritchardn
    pritchardn committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    d4b6e19 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/master' 

    # Conflicts:
#	daliuge-engine/test/reproducibility/test_pg_blockdag.py
    @pritchardn
    pritchardn committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    ff08e59 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2021

  1. Adds functionality to write reprodata out to file on a per-session ba… 

    …sis.
    @pritchardn
    pritchardn committed Aug 12, 2021
    Configuration menu
    Copy the full SHA
    1142ca3 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2021

  1. Adds GPUtil to requirements.txt 

    GPUtil is used when generating machine recomputation information.
    @pritchardn
    pritchardn committed Aug 19, 2021
    Configuration menu
    Copy the full SHA
    aecf2ef View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2022

  1. Merge remote-tracking branch 'upstream/master' into upstream_merge_ja… 

    …n_2022

# Conflicts:
#	Dockerfile
#	Dockerfile.ray
#	OpenAPI/tests/translator_test_client.py
#	build_dlg_ray.sh
#	daliuge-common/dlg/__init__.py
#	daliuge-common/dlg/clients.py
#	daliuge-common/dlg/common/__init__.py
#	daliuge-common/dlg/common/streams.py
#	daliuge-common/dlg/common/tool.py
#	daliuge-common/dlg/constants.py
#	daliuge-common/dlg/restutils.py
#	daliuge-common/setup.py
#	daliuge-engine/dlg/__init__.py
#	daliuge-engine/dlg/apps/__init__.py
#	daliuge-engine/dlg/apps/archiving.py
#	daliuge-engine/dlg/apps/bash_shell_app.py
#	daliuge-engine/dlg/apps/crc.py
#	daliuge-engine/dlg/apps/dockerapp.py
#	daliuge-engine/dlg/apps/mpi.py
#	daliuge-engine/dlg/apps/plasma.py
#	daliuge-engine/dlg/apps/pyfunc.py
#	daliuge-engine/dlg/apps/scp.py
#	daliuge-engine/dlg/apps/simple.py
#	daliuge-engine/dlg/apps/socket_listener.py
#	daliuge-engine/dlg/dask_emulation.py
#	daliuge-engine/dlg/deploy/common.py
#	daliuge-engine/dlg/deploy/dlg_monitor.py
#	daliuge-engine/dlg/deploy/dlg_proxy.py
#	daliuge-engine/dlg/deploy/pawsey/scale_test.py
#	daliuge-engine/dlg/deploy/pawsey/start_dfms_cluster.py
#	daliuge-engine/dlg/deploy/remotes.py
#	daliuge-engine/dlg/deploy/utils/monitor_replayer.py
#	daliuge-engine/dlg/drop.py
#	daliuge-engine/dlg/droputils.py
#	daliuge-engine/dlg/graph_loader.py
#	daliuge-engine/dlg/io.py
#	daliuge-engine/dlg/lifecycle/dlm.py
#	daliuge-engine/dlg/lifecycle/hsm/manager.py
#	daliuge-engine/dlg/lifecycle/hsm/store.py
#	daliuge-engine/dlg/lifecycle/registry.py
#	daliuge-engine/dlg/manager/cmdline.py
#	daliuge-engine/dlg/manager/composite_manager.py
#	daliuge-engine/dlg/manager/node_manager.py
#	daliuge-engine/dlg/manager/proc_daemon.py
#	daliuge-engine/dlg/manager/replay.py
#	daliuge-engine/dlg/manager/rest.py
#	daliuge-engine/dlg/manager/session.py
#	daliuge-engine/dlg/manager/web/dim.html
#	daliuge-engine/dlg/manager/web/dm.html
#	daliuge-engine/dlg/manager/web/session.html
#	daliuge-engine/dlg/manager/web/static/css/session.css
#	daliuge-engine/dlg/manager/web/static/js/dm.js
#	daliuge-engine/dlg/ngaslite.py
#	daliuge-engine/dlg/remote.py
#	daliuge-engine/dlg/rpc.py
#	daliuge-engine/dlg/runtime/__init__.py
#	daliuge-engine/dlg/runtime/tool_commands.py
#	daliuge-engine/dlg/testutils.py
#	daliuge-engine/dlg/utils.py
#	daliuge-engine/fabfile/APPspecific.py
#	daliuge-engine/fabfile/__init__.py
#	daliuge-engine/test/apps/test_bash.py
#	daliuge-engine/test/apps/test_crc.py
#	daliuge-engine/test/apps/test_docker.py
#	daliuge-engine/test/apps/test_dynlib.py
#	daliuge-engine/test/apps/test_dynlib2.py
#	daliuge-engine/test/apps/test_plasma.py
#	daliuge-engine/test/apps/test_pyfunc.py
#	daliuge-engine/test/apps/test_simple.py
#	daliuge-engine/test/deploy/test_common.py
#	daliuge-engine/test/deploy/test_slurm_utils.py
#	daliuge-engine/test/integrate/chiles/chilesdo.py
#	daliuge-engine/test/integrate/chiles/chilesdoapp.py
#	daliuge-engine/test/integrate/chiles/chilesdospec.py
#	daliuge-engine/test/integrate/chiles/chilesdospec_docker.py
#	daliuge-engine/test/integrate/example_split.py
#	daliuge-engine/test/integrate/freq_split.py
#	daliuge-engine/test/integrate/msconverter/ms_converter.py
#	daliuge-engine/test/lifecycle/test_dlm.py
#	daliuge-engine/test/lifecycle/test_registry.py
#	daliuge-engine/test/manager/test_daemon.py
#	daliuge-engine/test/manager/test_dim.py
#	daliuge-engine/test/manager/test_dm.py
#	daliuge-engine/test/manager/test_mm.py
#	daliuge-engine/test/manager/test_rest.py
#	daliuge-engine/test/manager/test_scalability.py
#	daliuge-engine/test/manager/testutils.py
#	daliuge-engine/test/memoryUsage.py
#	daliuge-engine/test/test_JsonDrop.py
#	daliuge-engine/test/test_dask_emulation.py
#	daliuge-engine/test/test_drop.py
#	daliuge-engine/test/test_droputils.py
#	daliuge-engine/test/test_graph_loader.py
#	daliuge-engine/test/test_io.py
#	daliuge-engine/test/test_session.py
#	daliuge-engine/test/test_tool.py
#	daliuge-engine/test/test_utils.py
#	daliuge-ray.yaml
#	daliuge-translator/dlg/__init__.py
#	daliuge-translator/dlg/dropmake/dm_utils.py
#	daliuge-translator/dlg/dropmake/pg_generator.py
#	daliuge-translator/dlg/dropmake/scheduler.py
#	daliuge-translator/dlg/dropmake/utils/antichains.py
#	daliuge-translator/dlg/dropmake/utils/bash_parameter.py
#	daliuge-translator/dlg/dropmake/web/Inspector_test.css
#	daliuge-translator/dlg/dropmake/web/Inspector_test.js
#	daliuge-translator/dlg/dropmake/web/go.js
#	daliuge-translator/dlg/dropmake/web/jquery.min.map
#	daliuge-translator/dlg/dropmake/web/jsoneditor.map
#	daliuge-translator/dlg/dropmake/web/lg_web.py
#	daliuge-translator/dlg/dropmake/web/matrix_vis.html
#	daliuge-translator/dlg/translator/tool_commands.py
#	daliuge-translator/setup.py
#	daliuge-translator/test/dropmake/logical_graphs/cont_img.graph
#	daliuge-translator/test/dropmake/test_lg_fill.py
#	daliuge-translator/test/dropmake/test_lgweb.py
#	daliuge-translator/test/dropmake/test_pg_gen.py
#	daliuge-translator/test/dropmake/test_scheduler.py
#	docs/conf.py
    @pritchardn
    pritchardn committed Jan 25, 2022
    Configuration menu
    Copy the full SHA
    f7e63be View commit details
    Browse the repository at this point in the history

  2. Adds compatibility for latest DALiuGE version

    @pritchardn
    pritchardn committed Jan 25, 2022
    Configuration menu
    Copy the full SHA
    4915289 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2022

  1. Moves reprodata logging to logdir.

    @pritchardn
    pritchardn committed Feb 8, 2022
    Configuration menu
    Copy the full SHA
    adf82ce View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2022

  1. Merges changes from the mainline March 2022 release

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    464fe00 View commit details
    Browse the repository at this point in the history

  2. Adds GPUtil to docs requirements.txt

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    6264ff2 View commit details
    Browse the repository at this point in the history

  3. Adds GPUtil to daliuge-engine

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    1d3c30c View commit details
    Browse the repository at this point in the history

  4. Adds GPUtil to requirements files.

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    50d4e0b View commit details
    Browse the repository at this point in the history

  5. Adds GPUtil to requirements files pt II.

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    333b543 View commit details
    Browse the repository at this point in the history

  6. Adds merklelib to requirements files.

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    44ec206 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'upstream_merge_jan_2022' into yan-919 

    # Conflicts:
#	docs/requirements.txt
    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    e0df4c7 View commit details
    Browse the repository at this point in the history

  8. Adds specific library versions for requirements

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    40c936b View commit details
    Browse the repository at this point in the history

  9. Adds gputil install step to yaml pipeline

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    78d7704 View commit details
    Browse the repository at this point in the history

  10. Adds merklelib install step to yaml pipeline

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    8c2eb88 View commit details
    Browse the repository at this point in the history

  11. Adds merklelib install step to yaml pipeline

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    b6ec44b View commit details
    Browse the repository at this point in the history

  12. Formatting to prevent docs warnings.

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    eb8fc12 View commit details
    Browse the repository at this point in the history

  13. Adds merklelib and gputil to setup.py scripts.

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    1d7fa0b View commit details
    Browse the repository at this point in the history

  14. Moves requirements to common instead of engine and translator.

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    4517bc4 View commit details
    Browse the repository at this point in the history

  15. Adds install requires command to setup.py.

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    d88bba9 View commit details
    Browse the repository at this point in the history

  16. Separates reproducibility tests into runtime / translator sections. 

    Adjusts test_scatter_blockdag.py to point to appropriate test graph location.
    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    5f18cbe View commit details
    Browse the repository at this point in the history

  17. Makes reprodata fetching in translator optional. Defaulting to {}

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    06f989c View commit details
    Browse the repository at this point in the history

  18. Updates test graph formats.

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    13b9da0 View commit details
    Browse the repository at this point in the history

  19. LG only appends reprodata if it was present in the original.

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    29ce80a View commit details
    Browse the repository at this point in the history

  20. Blocks run through pgt initalisation without lgt initialization are g… 

    …iven empty reprodata dictionaries.
    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    13a46c4 View commit details
    Browse the repository at this point in the history

  21. Pops attached reprodata when testing end to end

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    bcb7d8e View commit details
    Browse the repository at this point in the history

  22. test_pg_gen imports pkg_resources

    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    b6ca129 View commit details
    Browse the repository at this point in the history

  23. Reprodata when translating is now either a well-formed dictionary or … 

    …None.

The unroll operation only appends reprodata if it is not None.
    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    673ff60 View commit details
    Browse the repository at this point in the history

  24. Slight adjustment, dictionary will exist but will be empty if reproda… 

    …ta is not passed from lgt layer.
    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    e8f4705 View commit details
    Browse the repository at this point in the history

  25. Slight adjustment, dictionary will exist but will be empty if reproda… 

    …ta is not passed from lgt layer.
    @pritchardn
    pritchardn committed Mar 16, 2022
    Configuration menu
    Copy the full SHA
    761ed2a View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2022

  1. Implements separated provenance information extraction methods. 

    Includes a way to define operations on Provenance information (currently store and count).
    @pritchardn
    pritchardn committed Mar 17, 2022
    Configuration menu
    Copy the full SHA
    9217fa1 View commit details
    Browse the repository at this point in the history

  2. Updates lgt data accumulation and affected tests with new extraction … 

    …method.
    @pritchardn
    pritchardn committed Mar 17, 2022
    Configuration menu
    Copy the full SHA
    2b54063 View commit details
    Browse the repository at this point in the history

  3. Updates lg data accumulation and affected tests with new extraction m… 

    …ethod.
    @pritchardn
    pritchardn committed Mar 17, 2022
    Configuration menu
    Copy the full SHA
    a978ca3 View commit details
    Browse the repository at this point in the history

  4. Updates pgt data accumulation and affected tests with new extraction … 

    …method.
    @pritchardn
    pritchardn committed Mar 17, 2022
    Configuration menu
    Copy the full SHA
    49e8f66 View commit details
    Browse the repository at this point in the history

  5. Updates pg data accumulation and affected tests with new extraction m… 

    …ethod.
    @pritchardn
    pritchardn committed Mar 17, 2022
    Configuration menu
    Copy the full SHA
    1f0bdb0 View commit details
    Browse the repository at this point in the history

  6. Makes categoryType and category field fetches, 'get' calls. 

    There are some older test graphs which are missing CategoryType fields. While incorrect, this shouldn't affect the execution of the graph.
    @pritchardn
    pritchardn committed Mar 17, 2022
    Configuration menu
    Copy the full SHA
    692bc77 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2022

  1. Changes dag traversal error to a warning. 

    Makes signature building methods significantly more robust to missing fields.
    @pritchardn
    pritchardn committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    5ce0ff6 View commit details
    Browse the repository at this point in the history

  2. Updates test graph to use latest drop type names.

    @pritchardn
    pritchardn committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    56ed5d2 View commit details
    Browse the repository at this point in the history

  3. Adds reprodata to pgt class.

    @pritchardn
    pritchardn committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    f0f4946 View commit details
    Browse the repository at this point in the history

  4. unroll operation always appends logical graph reprodata, even if it i… 

    …s empty.
    @pritchardn
    pritchardn committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    3e099b8 View commit details
    Browse the repository at this point in the history

  5. Updates chiles_simple.graph to latest graphformat.

    @pritchardn
    pritchardn committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    b3bc9cf View commit details
    Browse the repository at this point in the history

  6. Adds signature construction to lg_web rest interface.

    @pritchardn
    pritchardn committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    27fea71 View commit details
    Browse the repository at this point in the history

  7. Adds missing partition signature functionality.

    @pritchardn
    pritchardn committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    d5702fd View commit details
    Browse the repository at this point in the history

  8. Adds empty case for init_lg_repro_data

    @pritchardn
    pritchardn committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    6622dc0 View commit details
    Browse the repository at this point in the history

  9. Adds empty graph init_repro_data tests.

    @pritchardn
    pritchardn committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    ed552ea View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2022

  1. Adds rmode to jsonbody_post method.

    @pritchardn
    pritchardn committed Mar 21, 2022
    Configuration menu
    Copy the full SHA
    d094077 View commit details
    Browse the repository at this point in the history

  2. Adds space for fields for remaining drop types

    @pritchardn
    pritchardn committed Mar 21, 2022
    Configuration menu
    Copy the full SHA
    b0e5296 View commit details
    Browse the repository at this point in the history

  3. Adds reproducibility fields for missing drop types

    @pritchardn
    pritchardn committed Mar 21, 2022
    Configuration menu
    Copy the full SHA
    44f9f01 View commit details
    Browse the repository at this point in the history

  4. Removes unnecessary import

    @pritchardn
    pritchardn committed Mar 21, 2022
    Configuration menu
    Copy the full SHA
    561f0ff View commit details
    Browse the repository at this point in the history

  5. Adds removed common_hash import

    @pritchardn
    pritchardn committed Mar 21, 2022
    Configuration menu
    Copy the full SHA
    cdb7da0 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2022

  1. Re-engages pgt blockdag building.

    @pritchardn
    pritchardn committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    8acec29 View commit details
    Browse the repository at this point in the history

  2. Creates load_lg function to return lg/t json dict.

    @pritchardn
    pritchardn committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    4c58a99 View commit details
    Browse the repository at this point in the history

  3. Adds lgt reprodata inits to gen_pgt (get and post) methods.

    @pritchardn
    pritchardn committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    bb152d8 View commit details
    Browse the repository at this point in the history

  4. lg_web logs reprodata now.

    @pritchardn
    pritchardn committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    114da1c View commit details
    Browse the repository at this point in the history

  5. pg_manager appends reprodata when passing for graph rendering.

    @pritchardn
    pritchardn committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    e40e91b View commit details
    Browse the repository at this point in the history

  6. Reprodata is logged to console.

    @pritchardn
    pritchardn committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    f6e8b08 View commit details
    Browse the repository at this point in the history

  7. Updates use of categoryType to category in STORAGE_TYPES when computi… 

    …ng block dags.
    @pritchardn
    pritchardn committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    be3b3e1 View commit details
    Browse the repository at this point in the history

Commits on Mar 24, 2022

  1. Adds category field to dummy drops in test_pg_blockdag.py

    @pritchardn
    pritchardn committed Mar 24, 2022
    Configuration menu
    Copy the full SHA
    faeeb0e View commit details
    Browse the repository at this point in the history

  2. Adds support for an ALL reproducibility mode.

    @pritchardn
    pritchardn committed Mar 24, 2022
    Configuration menu
    Copy the full SHA
    6f2de7b View commit details
    Browse the repository at this point in the history

  3. Adds test for ALL rmode support. And fixes preexisting tests.

    @pritchardn
    pritchardn committed Mar 24, 2022
    Configuration menu
    Copy the full SHA
    2330bc4 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2022

  1. Updates constants.py for pylint compliance. 

    Upgrades protocol to 1.0 in preparation for integration into main branch.
    @pritchardn
    pritchardn committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    e774e82 View commit details
    Browse the repository at this point in the history

  2. Adjusts reproducibility.py and reproducibility_fields.py for pylint c… 

    …ompatibility
    @pritchardn
    pritchardn committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    2fdac06 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2022

  1. Initial Attempt to add reprodata to MKN nodes.

    @pritchardn
    pritchardn committed Mar 28, 2022
    Configuration menu
    Copy the full SHA
    f114c04 View commit details
    Browse the repository at this point in the history

  2. Adds provisional MKN node support (allows for incorrect reprodata con… 

    …struction for now.)
    @pritchardn
    pritchardn committed Mar 28, 2022
    Configuration menu
    Copy the full SHA
    2cd3e50 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/yan-919' into yan-919

    @pritchardn
    pritchardn committed Mar 28, 2022
    Configuration menu
    Copy the full SHA
    aaf870c View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into yan-919 

    # Conflicts:
#	daliuge-engine/dlg/apps/dockerapp.py
#	daliuge-engine/dlg/apps/plasma.py
#	daliuge-engine/test/apps/test_plasma.py
    @pritchardn
    pritchardn committed Mar 28, 2022
    Configuration menu
    Copy the full SHA
    48e1e26 View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2022

  1. Formatting of library with Black

    @pritchardn
    pritchardn committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    db820d0 View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2022

  1. Hashing Algorithm string is now full.

    pritchardn committed Apr 4, 2022
    Configuration menu
    Copy the full SHA
    9d40564 View commit details
    Browse the repository at this point in the history

  2. Session object now supports reproducibilityflags.ALL mode

    @pritchardn
    pritchardn committed Apr 4, 2022
    Configuration menu
    Copy the full SHA
    f5e2196 View commit details
    Browse the repository at this point in the history

  3. lgweb appends reprodata

    @pritchardn
    pritchardn committed Apr 4, 2022
    Configuration menu
    Copy the full SHA
    bec738e View commit details
    Browse the repository at this point in the history

  4. graph_loader.py now looks for 'rmode' instead of 'merkleroot'

    @pritchardn
    pritchardn committed Apr 4, 2022
    Configuration menu
    Copy the full SHA
    660e51b View commit details
    Browse the repository at this point in the history

  5. drops are no longer automatically committed upon a repromode change. 

    Updates related tests.
    @pritchardn
    pritchardn committed Apr 4, 2022
    Configuration menu
    Copy the full SHA
    24bd2f3 View commit details
    Browse the repository at this point in the history

  6. Composite manager now attempts to add reprodata to all graph partitions.

    @pritchardn
    pritchardn committed Apr 4, 2022
    Configuration menu
    Copy the full SHA
    191af5e View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2022

  1. Adds a reprodata comparison script which can be run standalone or int… 

    …egrate programatically.
    @pritchardn
    pritchardn committed Apr 8, 2022
    Configuration menu
    Copy the full SHA
    b8cf0f6 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2022

  1. Recompute-data is generated in a best-effort basis (dynlib apps can m… 

    …is-behave).

File drops generate reproduce data on a best-effort basis.
Docker drops generate reprodata on completion
    @pritchardn
    pritchardn committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    637146b View commit details
    Browse the repository at this point in the history

  2. dylib recompute data is generated from a list (not a dictionary)

    @pritchardn
    pritchardn committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    d4b4863 View commit details
    Browse the repository at this point in the history

  3. Removes large debugging output from graph_loader

    @pritchardn
    pritchardn committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    986a20d View commit details
    Browse the repository at this point in the history

  4. Dumped reprodata includes original graph (may be reverted later once … 

    …stable).
    @pritchardn
    pritchardn committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    f425885 View commit details
    Browse the repository at this point in the history

  5. rflag_caster is more robust allowing for named conversion to flags.

    @pritchardn
    pritchardn committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    130d744 View commit details
    Browse the repository at this point in the history

  6. reprodata_compare.py updated to process directories full of multiple … 

    …files correctly.
    @pritchardn
    pritchardn committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    ea0b131 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2022

  1. Comparison script will only write an comparison if there is more than… 

    … a single workflow present.
    @pritchardn
    pritchardn committed Apr 13, 2022
    Configuration menu
    Copy the full SHA
    0fcda9e View commit details
    Browse the repository at this point in the history

  2. Makes reproduce mode only include the values of terminal nodes

    @pritchardn
    pritchardn committed Apr 13, 2022
    Configuration menu
    Copy the full SHA
    3c844f2 View commit details
    Browse the repository at this point in the history

  3. Removes recommit test since that operation does not raise an error an… 

    …ymore.
    @pritchardn
    pritchardn committed Apr 13, 2022
    Configuration menu
    Copy the full SHA
    32b2be7 View commit details
    Browse the repository at this point in the history

  4. Adjusts blockdag building tests when reproducing. Only terminal data … 

    …is considered.
    @pritchardn
    pritchardn committed Apr 13, 2022
    Configuration menu
    Copy the full SHA
    49e815b View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2022

  1. Adds docstring documentation to cli.rst and modifies CLI arg descript… 

    …ions in tool_commands.py.
    @pritchardn
    pritchardn committed Apr 14, 2022
    Configuration menu
    Copy the full SHA
    2558e9e View commit details
    Browse the repository at this point in the history

  2. Fully implements and tests per-session reprodata fetching from node m… 

    …anagers via REST interface.
    @pritchardn
    pritchardn committed Apr 14, 2022
    Configuration menu
    Copy the full SHA
    74caaab View commit details
    Browse the repository at this point in the history

  3. Adds test asserting equivalence between each individual rmode and the… 

    …ir 'all' counterpart.
    @pritchardn
    pritchardn committed Apr 14, 2022
    Configuration menu
    Copy the full SHA
    5a19d01 View commit details
    Browse the repository at this point in the history

  4. Corrects implementation of ALL mode, they now all match.

    @pritchardn
    pritchardn committed Apr 14, 2022
    Configuration menu
    Copy the full SHA
    a312e45 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2022

  1. Adjusts how dynlib drops pack their inputs. 

    Attempts to get field values rather than descriptions.
    @pritchardn
    pritchardn committed Apr 19, 2022
    Configuration menu
    Copy the full SHA
    97c80ed View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2022

  1. Merge branch 'master' into yan-919 

    # Conflicts:
#	daliuge-common/dlg/common/__init__.py
#	daliuge-engine/dlg/apps/pyfunc.py
#	daliuge-engine/dlg/apps/simple.py
#	daliuge-engine/dlg/drop.py
#	daliuge-engine/dlg/graph_loader.py
#	daliuge-engine/dlg/manager/web/static/js/dm.js
#	daliuge-engine/dlg/rpc.py
#	daliuge-engine/fabfile/APPspecific.py
#	daliuge-engine/test/apps/test_pyfunc.py
#	daliuge-engine/test/graphs/test_graphExecution.py
#	daliuge-translator/dlg/dropmake/scheduler.py
#	tools/xml2palette/xml2palette.py
    @pritchardn
    pritchardn committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    e7060da View commit details
    Browse the repository at this point in the history

  2. Updates pyfunc reproducibility recompute information to handle new ar… 

    …gument structure.
    @pritchardn
    pritchardn committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    ce56d45 View commit details
    Browse the repository at this point in the history

  3. Updates blockdag building to expect dicts for links.

    @pritchardn
    pritchardn committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    c9fe5ec View commit details
    Browse the repository at this point in the history

  4. Adjusts split blockdag test to remove whole graph assertion.

    @pritchardn
    pritchardn committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    0314215 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2022

  1. Adds to HelloWorld demo app to accept both pickled and unpickled data.

    @pritchardn
    pritchardn committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    d68a740 View commit details
    Browse the repository at this point in the history

  2. Makes get rg_data 'best effort' - nodes which are present but not ult… 

    …imately run (e.g. gathers) will not have any runtime data themselves.
    @pritchardn
    pritchardn committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    47a1ded View commit details
    Browse the repository at this point in the history

  3. Adds appclass to PYTHON_APP lg fields - this one slipped through acci… 

    …dentally
    @pritchardn
    pritchardn committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    3339a30 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2022

  1. Prevents argument parsing from evaluating empty strings with ast.

    @pritchardn
    pritchardn committed May 2, 2022
    Configuration menu
    Copy the full SHA
    87b76c5 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2022

  1. Minor adjustment to reprodata_compare.py script - writes comparison f… 

    …ile with 2 or more graphs
    @pritchardn
    pritchardn committed May 6, 2022
    Configuration menu
    Copy the full SHA
    b8167e9 View commit details
    Browse the repository at this point in the history

  2. Changes how Rflag enums are created. 

    Adds test for rflag serialization
Refactors rflag_caster
    @pritchardn
    pritchardn committed May 6, 2022
    Configuration menu
    Copy the full SHA
    83a2558 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into yan-919 

    # Conflicts:
#	daliuge-engine/dlg/deploy/helm_client.py
#	daliuge-engine/dlg/deploy/start_helm_cluster.py
#	daliuge-engine/test/deploy/test_helm_deploy.py
#	daliuge-translator/dlg/dropmake/web/lg_web.py
    @pritchardn
    pritchardn committed May 6, 2022
    Configuration menu
    Copy the full SHA
    905de37 View commit details
    Browse the repository at this point in the history

  4. Final (hopefully) formatting run with black

    @pritchardn
    pritchardn committed May 6, 2022
    Configuration menu
    Copy the full SHA
    0c31220 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into yan-919

    @pritchardn
    pritchardn authored May 6, 2022
    Configuration menu
    Copy the full SHA
    b89ae99 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into yan-919

    @pritchardn
    pritchardn authored May 6, 2022
    Configuration menu
    Copy the full SHA
    e2aefd4 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2022

  1. Changes behavour of ReproducibilityFlags.NOTHING to exclude all refer… 

    …ence to reproducibility information (a backwards compatibility consideration)
    @pritchardn
    pritchardn committed May 9, 2022
    Configuration menu
    Copy the full SHA
    c143079 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/yan-919' into yan-919

    @pritchardn
    pritchardn committed May 9, 2022
    Configuration menu
    Copy the full SHA
    79ce001 View commit details
    Browse the repository at this point in the history

  3. Minor bug fix

    @pritchardn
    pritchardn committed May 9, 2022
    Configuration menu
    Copy the full SHA
    76bf5b8 View commit details
    Browse the repository at this point in the history

  4. Improves blockdag building to handle drops without reprodata.

    @pritchardn
    pritchardn committed May 9, 2022
    Configuration menu
    Copy the full SHA
    28d1e55 View commit details
    Browse the repository at this point in the history

  5. Adjusts lg_blockdag tests to account for new NOTHING behaviour

    @pritchardn
    pritchardn committed May 9, 2022
    Configuration menu
    Copy the full SHA
    83bbed1 View commit details
    Browse the repository at this point in the history

Commits on May 12, 2022

  1. Merge branch 'master' into yan-919

    @pritchardn
    pritchardn authored May 12, 2022
    Configuration menu
    Copy the full SHA
    37e1181 View commit details
    Browse the repository at this point in the history