Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The integration of SasView #199

Merged
merged 50 commits into from
Sep 1, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
49bd1ea
Add SasView example_runScripts
AtomAnu Jul 18, 2019
519174c
Create parse file for SasView
AtomAnu Jul 26, 2019
7f6d3a9
Update prob_def_1.txt
AtomAnu Jul 31, 2019
eb3d8dd
Update parse_sasview_data.py
AtomAnu Aug 1, 2019
e1e3259
Further integration of SasView
AtomAnu Aug 2, 2019
8cda081
SasView Successful Integration
AtomAnu Aug 6, 2019
d29060b
Enable FitBenchmarking to fit SasView models using 'scipy.optimize.cu…
AtomAnu Aug 7, 2019
1f60683
Update misc.py
AtomAnu Aug 7, 2019
4c70d12
Some errors fixed in the case of using SasView models in scipy fitting
AtomAnu Aug 7, 2019
53113fa
Futher Integration of SasView
AtomAnu Aug 20, 2019
496f614
Update `example_runScripts_SasView.py` and add `sas/sascalc` to FitBe…
AtomAnu Aug 21, 2019
9ee4a31
Minor adjustments
AtomAnu Aug 21, 2019
5690ad5
Update test_func_def.py
AtomAnu Aug 22, 2019
7c3d303
Removal of the test data files and update on parse.py
AtomAnu Aug 22, 2019
0da9bae
Update parse.py
AtomAnu Aug 22, 2019
f45c1db
Update parse_fitbenchmark_data.py
AtomAnu Aug 22, 2019
5cd4ea9
Update test_parse_fitbenchmark_data.py
AtomAnu Aug 22, 2019
260eb4b
Unit tests file for `parse_sasview_data.py`
AtomAnu Aug 22, 2019
bcb1df4
Temporary condition statement added in `test_parse_sasview_data.py` t…
AtomAnu Aug 22, 2019
5d1f847
Add `sasview/tests/test_prepare_data.py`
AtomAnu Aug 22, 2019
be5c5e3
Update setup.py to install bumps and sasmodels
AtomAnu Aug 22, 2019
70fcfd6
Update setup.py to install lxml
AtomAnu Aug 22, 2019
dc84f6a
Update README.md
AtomAnu Aug 23, 2019
6253144
Minor changes for modules inside `sasview/func_def.py`
AtomAnu Aug 23, 2019
5a47874
Update example_runScripts.py
AtomAnu Aug 23, 2019
36c482f
Update README.md
AtomAnu Aug 23, 2019
623c019
Create README.md for `sas/sascalc`
AtomAnu Aug 23, 2019
1ba97e0
Update README.md
AtomAnu Aug 23, 2019
d3d74af
Update README.md
AtomAnu Aug 23, 2019
103387c
Change the log messages sent when fits fail
AtomAnu Aug 23, 2019
06f2ac5
Change the name of folders inside `fitbenchmarking/benchmark_problems`
AtomAnu Aug 23, 2019
00439a4
Delete .DS_Store
AtomAnu Aug 23, 2019
c6beacb
Delete .DS_Store
AtomAnu Aug 23, 2019
ae14f0f
Update test_misc.py
AtomAnu Aug 23, 2019
0b1f77f
Merge branch 'integratingSasView' of https://github.com/fitbenchmarki…
AtomAnu Aug 23, 2019
4d41c51
Update test_parse_sasview_data.py
AtomAnu Aug 23, 2019
64ad54c
Update test_parse_sasview_data.py
AtomAnu Aug 23, 2019
9b74b7e
Added a unit test file for `sasview/func_def.py`
AtomAnu Aug 23, 2019
4e623bf
Update test_parse_sasview_data.py
AtomAnu Aug 23, 2019
eedf076
Update fitbenchmark_data_functions.py
AtomAnu Aug 23, 2019
049cc5e
Change to def description
Anders-Markvardsen Aug 23, 2019
6caf380
Minor fixes
AtomAnu Aug 26, 2019
8c83520
Merge branch 'integratingSasView' of https://github.com/fitbenchmarki…
AtomAnu Aug 26, 2019
9d9eb18
remove some unused imports
Anders-Markvardsen Aug 29, 2019
328b657
updated doc for two of the example script, incluidng using [] to show…
Anders-Markvardsen Aug 29, 2019
85d7d9a
added more clear doc utility nist and fitbenchmarking files
Anders-Markvardsen Aug 29, 2019
92834a9
update doc for sasview example file
Anders-Markvardsen Aug 29, 2019
2875c80
fix tests
Anders-Markvardsen Aug 30, 2019
845f9db
More doc update and remove file
Anders-Markvardsen Sep 1, 2019
6a3e68b
Describe new sasview format and give description of all problem folders
Anders-Markvardsen Sep 1, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
16 changes: 13 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ FitBenchmarking is a tool for comparing different minimizers/fitting software ba


## Quick Start
[![Python 2.7](https://img.shields.io/badge/python-2.7-blue.svg)](https://www.python.org/downloads/release/python-2715/) will be needed for running/installing this. Instructions for setting up python are available [here](https://github.com/mantidproject/fitbenchmarking/wiki/Setting-up-Python).
[![Python 2.7](https://img.shields.io/badge/python-2.7-blue.svg)](https://www.python.org/downloads/release/python-2715/) is (currently) the Python version needed for running/installing FitBenchmarking. Instructions for setting up python are available [here](https://github.com/mantidproject/fitbenchmarking/wiki/Setting-up-Python).

For help of how to use the command line/terminal, click the hyperlink corresponding to your operating system: [Windows](https://red-dot-geek.com/basic-windows-command-prompt-commands/), [macOS](http://newsourcemedia.com/blog/basic-terminal-commands/) and [Ubuntu](https://hackingpress.com/basic-linux-commands/)

Expand All @@ -28,10 +28,10 @@ For help of how to use the command line/terminal, click the hyperlink correspond

The above step is done to ensure that the compatible version of docutils packages (version 0.12) is installed.

7. Finally, in this terminal, run `example_scripts/example_runScript.py`, located in the fitbenchmarking folder. This example script fit benchmarks Mantid using all the available minimizers. The resulting tables can be found in `example_scripts/results`.
7. Finally, in this terminal, run `example_scripts/example_runScript_mantid.py`, located in the fitbenchmarking folder. This example script fit benchmarks Mantid using all the available minimizers. The resulting tables can be found in `example_scripts/results`.

## FitBenchmarking Scipy
The `example_runScripts.py` file can be changed such that it benchmarks minimizers supported by scipy instead of mantid (details provided in the file itself).
The `example_runScripts.py` file is designed to benchmark minimizers supported by software/libraries that provide straightforward cross-platform Python install; as of know this mean SciPy (more details provided in the file itself).

For this to work scipy version 0.17 or higher is needed (which includes needed [curve_fit](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html) support). **The Linux distributions we have tested against so far have all included scipy 0.17+ (0.17 is from Feb 2016).**

Expand All @@ -54,6 +54,16 @@ Mantid on Windows is shipped with Python. The above steps can also be done from
terminal, in which case please ensure that you are upgrading against Python
installed with Mantid, which by default is located in `C:\MantidInstall\bin`.

## FitBenchmarking SasView
The `example_runScripts_SasView.py` file is designed to benchmark minimizers supported by SasView (Bumps).

In order to do so, Bumps, sasmodels, lxml and sascalc need to be installed. Note that Bumps, sasmodels and lxml can be installed via `pip` commands. However, as of this writing, sascalc is not an independent package yet and, therefore, cannot be installed via `pip`. Thus, sascalc is now included in FitBenchmarking under the folder `fitbenchmarking/sas`.

To install Bumps, sasmodels and lxml, run the following command on console:
1. `python -m pip install bumps`
2. `python -m pip install sasmodels`
3. `python -m pip install lxml`

## Description
The tool creates a table/tables that shows a comparison between the different minimizers available in a fitting software (e.g. scipy or mantid), based on their accuracy and/or runtimes.
An example of a table is:
Expand Down
11 changes: 10 additions & 1 deletion benchmark_problems/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,14 @@ Two problem definition file formats are supported:

* The format native to FitBenchmarking (denoted FitBenchmark). For example, see the neutron_data folder.
* NIST Nonlinear Regression [format](https://www.itl.nist.gov/div898/strd/nls/data/LINKS/DATA/Misra1a.dat). For example, see the NIST folder.
* Format named sasview to benchmark against SAS-models used by SasView. See the SAS_modelling folder.

A data file name specified within FitBenchmark definition file is recommended to be stored in a sub-folder named `data_files` relative to the location of the definition file.
A data file name specified within a definition file is recommended to be stored in a sub-folder named `data_files` relative to the location of the definition file.

As of this writing the problem folders are:

* SAS_modelling : fitting problems specific relevant to fitting SAS (Small Angle Scattering) data
* Muon : generic fitting problems relevant to fitting Muon data collected at a Muon facility such as ISIS Neutron and Muon Facility
* Neutron : generic fitting problems relevant to fitting Neutron data collected at a Neutron facility such as ISIS Neutron and Muon Facility
* NIST : set of made up fitting problem (not against measured data with error bars) as described [here](https://www.itl.nist.gov/div898/strd/nls/nls_main.shtml)
* CUTEes : fitting problems relevant to this tool included in [CUTEet](http://epubs.stfc.ac.uk/bitstream/9327/RAL-TR-2013-005.pdf)
22 changes: 22 additions & 0 deletions benchmark_problems/SAS_modelling/1D/data_files/cyl_400_20.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<X> <Y>
0 -1.#IND
0.025 125.852
0.05 53.6662
0.075 26.0733
0.1 11.8935
0.125 4.61714
0.15 1.29983
0.175 0.171347
0.2 0.0417614
0.225 0.172719
0.25 0.247876
0.275 0.20301
0.3 0.104599
0.325 0.0285595
0.35 0.00213344
0.375 0.0137511
0.4 0.0312374
0.425 0.0350328
0.45 0.0243172
0.475 0.00923067
0.5 0.00121297
56 changes: 56 additions & 0 deletions benchmark_problems/SAS_modelling/1D/data_files/cyl_400_40.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
<X> <Y>
0 -1.#IND
0.00925926 1246.59
0.0185185 612.143
0.0277778 361.142
0.037037 211.601
0.0462963 122.127
0.0555556 65.2385
0.0648148 30.8914
0.0740741 12.4737
0.0833333 3.51371
0.0925926 0.721835
0.101852 0.583607
0.111111 1.31084
0.12037 1.9432
0.12963 1.94286
0.138889 1.58912
0.148148 0.987076
0.157407 0.456678
0.166667 0.147595
0.175926 0.027441
0.185185 0.0999575
0.194444 0.198717
0.203704 0.277667
0.212963 0.288172
0.222222 0.220056
0.231481 0.139378
0.240741 0.0541106
0.25 0.0140158
0.259259 0.0132187
0.268519 0.0336301
0.277778 0.0672911
0.287037 0.0788983
0.296296 0.0764438
0.305556 0.0555445
0.314815 0.0280548
0.324074 0.0111798
0.333333 0.00156156
0.342593 0.00830883
0.351852 0.0186266
0.361111 0.0275426
0.37037 0.03192
0.37963 0.0255329
0.388889 0.0175216
0.398148 0.0073075
0.407407 0.0016631
0.416667 0.00224153
0.425926 0.0051335
0.435185 0.0112914
0.444444 0.0138209
0.453704 0.0137453
0.462963 0.0106682
0.472222 0.00532472
0.481481 0.00230646
0.490741 0.000335344
0.5 0.00177224
6 changes: 6 additions & 0 deletions benchmark_problems/SAS_modelling/1D/prob_def_1.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# An example data set for SasView 1D data
name = 'Problem Def 1'
input_file = 'cyl_400_20.txt'
function ='name=cylinder,radius=35.0,length=350.0,background=0.0,scale=1.0,sld=4.0,sld_solvent=1.0'
parameter_ranges = 'radius.range(1,50);length.range(1,500)'
description = ''
6 changes: 6 additions & 0 deletions benchmark_problems/SAS_modelling/1D/prob_def_2.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# An example data set for SasView 1D data
name = 'Problem Def 2'
input_file = 'cyl_400_40.txt'
function ='name=cylinder,radius=35.0,length=350.0,background=0.0,scale=1.0,sld=4.0,sld_solvent=1.0'
parameter_ranges = 'radius.range(1,50);length.range(1,500)'
description = ''
10 changes: 5 additions & 5 deletions example_scripts/example_runScripts.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
from results_output import save_results_tables as printTables

# SPECIFY THE SOFTWARE/PACKAGE CONTAINING THE MINIMIZERS YOU WANT TO BENCHMARK
software = 'scipy'
software = ['scipy']
software_options = {'software': software}

# User defined minimizers
Expand All @@ -51,6 +51,7 @@
benchmark_probs_dir = os.path.join(fitbenchmarking_folder,
'benchmark_problems')


"""
Modify results_dir to specify where the results of the fit should be saved
If left as None, they will be saved in a "results" folder in the working dir
Expand All @@ -73,17 +74,16 @@

# ADD WHICH PROBLEM SETS TO TEST AGAINST HERE
# Do this, in this example file, by selecting sub-folders in benchmark_probs_dir
# "Muon_data" works for mantid minimizers
# problem_sets = ["Neutron_data", "NIST/average_difficulty"]
# problem_sets = ["CUTEst", "Muon_data", "Neutron_data", "NIST/average_difficulty", "NIST/high_difficulty", "NIST/low_difficulty"]
# problem_sets = ["CUTEst", "Muon", "Neutron", "NIST/average_difficulty", "NIST/high_difficulty", "NIST/low_difficulty",
# "SAS_modelling/1D"]

problem_sets = ["NIST/low_difficulty"]

for sub_dir in problem_sets:
# generate group label/name used for problem set
label = sub_dir.replace('/', '_')

# Problem data directory
# Create full path for the directory that holds a group of problem definition files
data_dir = os.path.join(benchmark_probs_dir, sub_dir)

test_data = glob.glob(data_dir + '/*.*')
Expand Down
125 changes: 125 additions & 0 deletions example_scripts/example_runScripts_SasView.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@


from __future__ import (absolute_import, division, print_function)
import os
import sys

# Avoid reaching the maximum recursion depth by setting recursion limit
# This is useful when running multiple data set benchmarking
# Otherwise recursion limit is reached and the interpreter throws an error
sys.setrecursionlimit(10000)

# Insert path to where the scripts are located, relative to
# the example_scripts folder
current_path = os.path.dirname(os.path.realpath(__file__))
fitbenchmarking_folder = os.path.abspath(os.path.join(current_path, os.pardir))
scripts_folder = os.path.join(fitbenchmarking_folder, 'fitbenchmarking')
sys.path.insert(0, scripts_folder)
sys.path.insert(1, fitbenchmarking_folder)

try:
import bumps
except:
print('******************************************\n'
'Bumps is not yet installed on your computer\n'
'To install, type the following command:\n'
'python -m pip install bumps\n'
'******************************************')
sys.exit()

try:
import sasmodels.data
except:
print('******************************************\n'
'sasmodels is not yet installed on your computer\n'
'To install, type the following command:\n'
'python -m pip install sasmodels\n'
'******************************************')
sys.exit()

try:
import sas
except:
print('******************************************\n'
'sas is not yet installed on your computer\n'
'To install, clone a version of SasView from https://github.com/SasView/sasview\n'
'After that, copy a folder called "sas" inside the sub-folder sasview/src to the fitbenchmarking directory\n'
'******************************************')
sys.exit()

from fitting_benchmarking import do_fitting_benchmark as fitBenchmarking
from results_output import save_results_tables as printTables

# SPECIFY THE SOFTWARE/PACKAGE CONTAINING THE MINIMIZERS YOU WANT TO BENCHMARK
software = ['sasview']
software_options = {'software': software}

# User defined minimizers
custom_minimizers = {"mantid": ["BFGS", "Simplex"],
"scipy": ["lm", "trf", "dogbox"],
"sasview": ["amoeba"]}
# custom_minimizers = None
# "amoeba", "lm", "newton", "de", "pt", "mp"

# SPECIFY THE MINIMIZERS YOU WANT TO BENCHMARK, AND AS A MINIMUM FOR THE SOFTWARE YOU SPECIFIED ABOVE
if len(sys.argv) > 1:
# Read custom minimizer options from file
software_options['minimizer_options'] = current_path + sys.argv[1]
elif custom_minimizers:
# Custom minimizer options:
software_options['minimizer_options'] = custom_minimizers
else:
# Using default minimizers from
# fitbenchmarking/fitbenchmarking/minimizers_list_default.json
software_options['minimizer_options'] = None


# Benchmark problem directories
benchmark_probs_dir = os.path.join(fitbenchmarking_folder,
'benchmark_problems')

"""
Modify results_dir to specify where the results of the fit should be saved
If left as None, they will be saved in a "results" folder in the working dir
If the full path is not given results_dir is created relative to the working dir
"""
results_dir = None

# Whether to use errors in the fitting process
use_errors = True

# Parameters of how the final tables are colored
# e.g. lower that 1.1 -> light yellow, higher than 3 -> dark red
# Change these values to suit your needs
color_scale = [(1.1, 'ranking-top-1'),
(1.33, 'ranking-top-2'),
(1.75, 'ranking-med-3'),
(3, 'ranking-low-4'),
(float('nan'), 'ranking-low-5')]

# ADD WHICH PROBLEM SETS TO TEST AGAINST HERE
# Do this, in this example file, by selecting sub-folders in benchmark_probs_dir
# problem_sets = ["CUTEst", "Muon", "Neutron", "NIST/average_difficulty", "NIST/high_difficulty", "NIST/low_difficulty",
# "SAS_modelling/1D"]
problem_sets = ["SAS_modelling/1D"]

for sub_dir in problem_sets:
# generate group label/name used for problem set
label = sub_dir.replace('/', '_')

# Create full path for the directory that holds a group of problem definition files
data_dir = os.path.join(benchmark_probs_dir, sub_dir)

print('\nRunning the benchmarking on the {} problem set\n'.format(label))
results_per_group, results_dir = fitBenchmarking(group_name=label, software_options=software_options,
data_dir=data_dir,
use_errors=use_errors, results_dir=results_dir)

print('\nProducing output for the {} problem set\n'.format(label))
for idx, group_results in enumerate(results_per_group):
# Display the runtime and accuracy results in a table
printTables(software_options, group_results,
group_name=label, use_errors=use_errors,
color_scale=color_scale, results_dir=results_dir)

print('\nCompleted benchmarking for {} problem set\n'.format(sub_dir))
6 changes: 3 additions & 3 deletions example_scripts/example_runScripts_expert.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
from resproc import visual_pages

# SPECIFY THE SOFTWARE/PACKAGE CONTAINING THE MINIMIZERS YOU WANT TO BENCHMARK
software = ['mantid', 'scipy']
software = ['scipy']
software_options = {'software': software}

# User defined minimizers
Expand Down Expand Up @@ -80,12 +80,12 @@
# Do this, in this example file, by selecting sub-folders in benchmark_probs_dir
# "Muon_data" works for mantid minimizers
# problem_sets = ["Neutron_data", "NIST/average_difficulty"]
problem_sets = ["Neutron_data"]
problem_sets = ["CUTEst"]
for sub_dir in problem_sets:
# generate group group_name/name used for problem set
group_name = sub_dir.replace('/', '_')

# Problem data directory
# Create full path for the directory that holds a group of problem definition files
data_dir = os.path.join(benchmark_probs_dir, sub_dir)

test_data = glob.glob(data_dir + '/*.*')
Expand Down
15 changes: 7 additions & 8 deletions example_scripts/example_runScripts_mantid.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,13 @@
from results_output import save_results_tables as printTables

# SPECIFY THE SOFTWARE/PACKAGE CONTAINING THE MINIMIZERS YOU WANT TO BENCHMARK
software = 'mantid'
software = ["mantid"]
software_options = {'software': software}

# User defined minimizers
# custom_minimizers = {"mantid": ["BFGS", "Simplex"],
# "scipy": ["lm", "trf", "dogbox"]}
custom_minimizers = None
custom_minimizers = {"mantid": ["Simplex"],
"scipy": ["lm", "trf", "dogbox"]}
# custom_minimizers = None


# SPECIFY THE MINIMIZERS YOU WANT TO BENCHMARK, AND AS A MINIMUM FOR THE SOFTWARE YOU SPECIFIED ABOVE
Expand Down Expand Up @@ -84,15 +84,14 @@

# ADD WHICH PROBLEM SETS TO TEST AGAINST HERE
# Do this, in this example file, by selecting sub-folders in benchmark_probs_dir
# "Muon_data" works for mantid minimizers
# problem_sets = ["CUTEst", "Muon_data", "Neutron_data", "NIST/average_difficulty", "NIST/high_difficulty", "NIST/low_difficulty"]
problem_sets = ['Muon_data', 'CUTEst']
problem_sets = ["CUTEst", "Muon", "Neutron", "NIST/average_difficulty", "NIST/high_difficulty", "NIST/low_difficulty",
"SAS_modelling/1D"]

for sub_dir in problem_sets:
# generate group label/name used for problem set
label = sub_dir.replace('/', '_')

# Problem data directory
# Create full path for the directory that holds a group of problem definition files
data_dir = os.path.join(benchmark_probs_dir, sub_dir)

test_data = glob.glob(data_dir + '/*.*')
Expand Down