[WIP] Elasticsearch 2.x #448

Merged
merged 381 commits into from Jun 14, 2016

Projects

None yet

6 participants

@oalders
Member
oalders commented Apr 15, 2016

This is the branch which we will merge into production at QAH.

mickeyn and others added some commits Apr 13, 2016
@mickeyn mickeyn move scripts to server-side files 992cb30
@mickeyn mickeyn point cpanfile to latest ElasticSearchX::Model version a91bf58
@oalders oalders Merge pull request #445 from mickeyn/mickey/es_2_3
[WIP] some hacks to allow indexing on ES 2.3
acb20b5
@oalders oalders Merge branch 'master' into mi/es2
d2c3010
@oalders oalders Adds t/lib to path in bin/prove. 40c6843
@oalders
Member
oalders commented Apr 16, 2016

The merge broke a bunch of stuff. I'm trying to fix it, but I'd hold off on pulling in any changes locally until that happens.

oalders and others added some commits Apr 16, 2016
@oalders oalders Updates deps.
99e49ec
@oalders oalders Replace lazy_build in release model. eaf2eba
@oalders oalders Fix some ordering issues (deep recursion) in Release model. 6ab3be4
@oalders oalders Tidy.
3dd4d7b
@oalders oalders Debug Travis module installs.
afa6dd4
@oalders oalders Downgrades ExtUtils::HasCompiler to 0.012
8d17d46
@mickeyn mickeyn attribute not used since #d9d9bdb6 258989c
@mickeyn mickeyn esx_model is used in BUILD... can be default 46edaea
@mickeyn mickeyn lazy_build --> lazy + builder (+ tidy) 3ddea80
@mickeyn mickeyn slop: a WIP solution d4499c2
@mickeyn mickeyn don't seem to be used
c6bdefa
@mickeyn mickeyn cleanup
bb2971e
@mickeyn mickeyn Revert "esx_model is used in BUILD... can be default"
This reverts commit 46edaea.
1a7a689
@mickeyn mickeyn esx_model: lazy + builder
259f6ef
@mickeyn mickeyn use MetaCPAN::Types
9e491b8
@oalders oalders Merge pull request #451 from CPAN-API/mickey/es2_no_lazy_build
mickey/es2 no lazy build
281b31f
@oalders oalders s/after_script/after_failure/ 1309584
@haarg @oalders haarg fix provides data in snapshot
ba80a80
@oalders oalders Adds basic docs on logger config. deb2ee3
@oalders oalders Adds a role for building a config. 1fde2f7
@oalders oalders Get minion DSN from config file. 6d61005
@oalders oalders Use HasConfig role in Script role.
57dc4f2
@oalders oalders Tidy
a451b95
@oalders oalders Adds Test::Vars to cpanfile.
1661592
@oalders oalders Upgrades Search::Elasticsearch to 2.02
c22ac05
@oalders oalders Wrap call to _es_home with try/catch block. 615af97
@oalders oalders Tweak es_client building in test suite.
51327c6
@mickeyn mickeyn WIP: Types cleanup e431f05
@mickeyn mickeyn removed redundant `required => 0` 3869113
@mickeyn mickeyn unused predicates 11e2ab3
@mickeyn mickeyn no rw attributes 8154088
@mickeyn mickeyn added missing module for test
8a913f3
@mickeyn mickeyn Merge pull request #453 from CPAN-API/mickey/es2_types_cleanup
WIP: cleaning up types usage
1a7e606
@mickeyn mickeyn few more fixes for problems exposed by tests
6077bf3
@oalders oalders Default prove wrapper to use test ES port. 86543a4
@oalders oalders Adds setup method to TestServer.pm
4c18905
@oalders oalders Updates cpanfile.snapshot
809227f
@mickeyn mickeyn missing 1
a337324
@oalders oalders Adds docs for dependency management. cb78d01
@oalders oalders Allow explicit undef in return. b5d7ecd
@oalders oalders Move MetaCPAN::Document::File::Set out of inner package.
21cf54a
@oalders oalders Adds basic docs about user authentication. cf7e497
@oalders oalders Comment out after_failure in Travis config. 383efec
@oalders oalders Tidy
bfffc9d
@oalders oalders s/carton-exec/run/
98c240c
@ranguard ranguard bulk_index no work, use bulk_helper instead
24b9215
@oalders oalders Merge pull request #454 from CPAN-API/leo/switch_builk_to_new_es
bulk_index no work, use bulk_helper instead
f971482
@oalders oalders Upgrades ESX::Model to 1.0.1
792c441
@ranguard ranguard fixup tickets script, skip SSL verify and add pod 709cfc5
@mickeyn mickeyn fix test for parsing POD errors
5f3d4fc
@ranguard ranguard make perlcritic config less... critical 4134c89
@ranguard ranguard convert ratings to bulk_helper
93ae2cf
@oalders oalders Merge pull request #456 from CPAN-API/leo/make_scripts_work
Leo/make scripts work
44fb92d
@ranguard ranguard switch sesson script to new ES bulk_helper
0452ae9
@oalders oalders Merge pull request #457 from CPAN-API/leo/convert_session
switch sesson script to new ES bulk_helper
f95d58c
@ranguard ranguard strip out invalid lat/lon values
43551cf
@oalders oalders Merge pull request #459 from CPAN-API/leo/cleanup_auth_data
strip out invalid lat/lon values
618607e
@mickeyn mickeyn removed content_cb: not really used, no point in having lazy content 366764c
@mickeyn mickeyn some attributes should be 'required' to get indexed
0e28172
@mickeyn mickeyn test fixes
e6043f7
@mickeyn mickeyn untangle `abstract` and `document` builders.
use a new lazy attribute for `section` and remove the direct
dependency loop between `abstract` and `document`.
dbc1a71
@mickeyn mickeyn names and types cleanup f57b47b
@mickeyn mickeyn refactoring import_archive 53dd022
@mickeyn mickeyn fix test: release/packages
6859a16
@oalders oalders Upgrade ESX::Model.
2a5c76e
@oalders oalders Fixes some quotes. 9cf86b3
@oalders oalders Use ternary rather than 2 returns.
f20a101
@mickeyn mickeyn we suspect this is just wrong b2264f7
@mickeyn mickeyn test fix: document setting logic is correct
7b2c1ee
@mickeyn mickeyn attribute corrections for documents
35b6ba5
@oalders oalders set_associated_pod() no longer wants a $file
61ca678
@oalders oalders ValuesAndExpressions::ProhibitAccessOfPrivateData has false positives.
6da9aa9
@mickeyn mickeyn only use Cpanel::JSON::XS 37fa67e
@mickeyn mickeyn fix test document/module 309718e
@mickeyn mickeyn fix test release/badpod
1b1d79d
@mickeyn mickeyn fix test release/packages-unclaimable 9ff78d7
@mickeyn mickeyn fix test release/no-modules 40e129c
@mickeyn mickeyn fix test release/no-packages
3d91364
@mickeyn mickeyn fixes few tests
04f4733
@mickeyn mickeyn some missed changes to Cpanel::JSON::XS
e0d2e9d
@oalders oalders Document how to get Minion job and worker stats.
3e38047
@jberger jberger Added script which fetches and indexes river data
closes #460
873522e
@oalders oalders Merge pull request #462 from CPAN-API/import_river_data
Added script which fetches and indexes river data
2015015
@oalders oalders Adds MetaCPAN::Moose to cpanfile
05b54c4
@mickeyn mickeyn fix test release/binary-data 52a83c9
@mickeyn mickeyn query fixes cd44d6a
@mickeyn mickeyn underlying module needs this ENV setting
cc3e519
@mickeyn mickeyn correct 'indexed' field for all files in archive
8a839eb
@oalders oalders Try to have Travis use Elasticsearch 2.3.0 76ef8f2
@oalders oalders Adds minimal docs about indexing.
56901b5
@oalders oalders Revert "Try to have Travis use Elasticsearch 2.3.0"
This reverts commit 76ef8f2.
84f2ae2
@oalders oalders Revert "Revert "Try to have Travis use Elasticsearch 2.3.0""
This reverts commit 84f2ae2.
12ad29e
@oalders oalders Follow redirects when having Travis download Elasticsearch deb.
cc74ce6
@mickeyn mickeyn fix test release/file-duplicates 3d9c499
oalders and others added some commits May 5, 2016
@oalders oalders numify_version already returns 0 if version isn't a true value.
4c41513
@oalders oalders Explicitly import catch/try.
8631fa1
@oalders oalders Make version default more succinct. 7dcdf8c
@oalders oalders Move version_numified from builder to default.
fcf3166
@oalders oalders Merge pull request #464 from CPAN-API/leo/cleanup_mirrors
Cleanup PC policy, fix mirrors to delete first #463
47349eb
@oalders oalders Merge pull request #468 from CPAN-API/encoded_river
Accept gzipped river data
9513146
@oalders oalders Bumps version of Code::TidyAll.
5e86fe1
@mickeyn mickeyn fix 'first' setting
don't default to 'true' for 'first', wait till later in the
release entry creation and then check it.
(otherwise we get multiple 'first' releases per distribution
if we index them together)
892cebf
@mickeyn mickeyn use match_phrase. fix test release/moose
in document matching when we want to keep the order of words,
we need to use `match_phrase` instead of `match`.
f7fd585
@mickeyn mickeyn fix test t/server/controller/author.t
502525a
@oalders oalders Possibly temporary fix to versions in CPANTesters import.
d92ad01
@oalders oalders Revert "Possibly temporary fix to versions in CPANTesters import."
This reverts commit d92ad01.
3c0c4eb
@ranguard ranguard drop invalid author lat/lon info
7bd8d01
@oalders oalders Merge pull request #472 from CPAN-API/leo/drop_duff_author_geo_info
drop invalid author lat/lon info
a41f506
@ranguard ranguard Validate author info is correct format
810a462
@oalders oalders Merge pull request #473 from CPAN-API/leo/make_author_stricter
Validate author info is correct format
05d7afe
@oalders oalders Revert "Revert "Possibly temporary fix to versions in CPANTesters imp…
…ort.""

This reverts commit 3c0c4eb.
44f146a
@oalders oalders Don't perform regex on undef value. 2031c63
@oalders oalders Minor cleanup. 0aa1b74
@oalders oalders CPANTesters script needs to search on the release type, not _every_ t…
…ype.
9fe8662
@oalders oalders Don't alter the version when checking it. 8fe1967
@oalders oalders s/bulk/insert_ok/ c47dad6
@oalders oalders You can now index a single release when testing.
1488c64
@oalders oalders Not every user is an author. a72cd2f
@oalders oalders Only prepare user test data once. a832cf8
@oalders oalders Remove debugging.
2793c67
@mickeyn mickeyn add asciiname to the author index
08ba3b8
@mickeyn mickeyn allow empty asciiname for author so all records get into ES
340bac6
@mickeyn mickeyn prevent race-condition for 'first' setting in testing mode
5d617ae
@mickeyn mickeyn make it less warny
8e55db0
@mickeyn mickeyn rework set_first_release + add first script to tests
this change will prevent the race between setting all releases to
first=0 and then setting first=1 based on a query that depends
on previous put to finish on ES level.

it adds a run of MetaCPAN::Script::First to the setup test,
to ensure all releases are indexed correctly ('first'-wise)
before running the other tests.
e87aca6
@oalders oalders Merge pull request #474 from CPAN-API/mickey/rework_set_first
rework set_first_release + add first script to tests
da9118c
@oalders oalders Add a UA string to river script so that NEILB's web server doesn't re…
…ject it.
de142a2
@mickeyn mickeyn version restriction is not needed
carton installs a later version (1.20) anyway because of some other dependency.
f00fc15
@oalders oalders CPANTesters db returns an empty string rather than 0 for versions.
a047d09
@oalders oalders Strip some characters from versions before looking up dists for CPANT…
…esters data.
a7b0856
@oalders oalders SKIP testers data check in ipsonar-0.29.t
5e07e65
@mickeyn mickeyn call prepare_user_test_data directly from the setup test 0cb3978
@mickeyn mickeyn Account: fix queries 669630f
@mickeyn mickeyn fix test server/controller/user/favorite
161c986
@mickeyn mickeyn add missing writer 5f0142b
@mickeyn mickeyn looks_human -> required (to force build when creating the user object)
6c384f1
@mickeyn mickeyn fix warning
1. case-insensitive extension in the 2nd case (like 1st & 3rd)
2. read from correct key in %_pod_score
fc3666b
@ranguard ranguard add a script for munin to use to monitor the queue
d63af5b
@mickeyn mickeyn add warnings to queue results
095a93c
@oalders oalders Merge pull request #479 from CPAN-API/mickey/queue_warnings
add warnings to queue results
912021b
@mickeyn mickeyn Only add warnings key to 'result' if there are warnings
39afb0d
@ranguard ranguard Merge pull request #480 from CPAN-API/mickey/simplify_warnings_querying
Only add warnings key to 'result' if there are warnings
c28912f
@ranguard ranguard add exec bit to monitor script
17a1669
@ranguard ranguard munin is not patient, hard code config
18080fa
@ranguard ranguard update minion queue monitoring further
c0f7664
@mickeyn mickeyn fix backup script (use update when id exists)
432e730
@oalders oalders Merge pull request #481 from CPAN-API/mickey/fix_backup
fix backup script (use update when id exists)
c022205
@mickeyn mickeyn prompt for confirmation on mapping --delete
64f0232
@ranguard ranguard Merge pull request #482 from CPAN-API/mickey/prompt4delete
prompt for confirmation on mapping --delete
4aa8361
@mickeyn mickeyn Fix 'documentation' mapping
This fix will change the 'documentation' mapping from:

    "documentation" : {
          "type" : "string"
    }

to:
    "documentation" : {
        "fields" : {
           "camelcase" : {
              "store" : true,
              "analyzer" : "camelcase",
              "type" : "string"
           },
           "lowercase" : {
              "analyzer" : "lowercase",
              "type" : "string",
              "store" : true
           },
           "edge" : {
              "analyzer" : "edge",
              "type" : "string",
              "store" : true
           },
           "analyzed" : {
              "analyzer" : "standard",
              "type" : "string",
              "fielddata" : {
                 "format" : "disabled"
              },
              "store" : true
           },
           "edge_camelcase" : {
              "store" : true,
              "type" : "string",
              "analyzer" : "edge_camelcase"
           }
        },
        "ignore_above" : 2048,
        "index" : "not_analyzed",
        "type" : "string"
     }

This change will also restore the original behavior of `autocomplete`
and fix the test t/server/controller/search/autocomplete.t
It is also needed to fix a test in the metacpan-web repo.
3350fe3
@oalders oalders Merge pull request #484 from CPAN-API/mickey/fix_documentation_mapping
Fix 'documentation' mapping
3a5c168
@mickeyn mickeyn updating bulk code for watcher script
9965142
@ranguard ranguard Merge pull request #485 from CPAN-API/mickey/test_watcher
updating bulk code for watcher script
72a38e9
@mickeyn mickeyn Put description & abstract info into ES
I can't recall why we removed it, but it is now missing
in metacpan-web (+tests)
0bfdc75
@ranguard ranguard Merge pull request #486 from CPAN-API/mickey/add_description_and_abst…
…ract_to_es

Put description & abstract info into ES
f0e0ff0
@coveralls
coveralls commented May 24, 2016 edited

Coverage Status

Coverage increased (+1.4%) to 90.23% when pulling f0e0ff0 on mi/es2 into 480d931 on master.

mickeyn and others added some commits May 27, 2016
@mickeyn mickeyn t/model/archive: replace size check with content check
0cb288b
@oalders oalders Merge pull request #487 from CPAN-API/mickey/t_model_archive
t/model/archive: replace size check with content check
d6573f9
@coveralls
coveralls commented May 27, 2016 edited

Coverage Status

Coverage increased (+1.07%) to 89.902% when pulling d6573f9 on mi/es2 into 480d931 on master.

@mickeyn mickeyn upgrade Perl to 5.22
0ab5d84
@oalders
Member

We can delete all Perls < 5.22 from the matrix, since we expect that anyone who deploys the this repo will be doing it via puppet.

mickeyn and others added some commits May 27, 2016
@mickeyn mickeyn remove older Perls from matrix
9b51fd3
@mickeyn mickeyn t/model/archive: replace size check with content check ddf8cee
@mickeyn mickeyn snapshot updates
9d43c17
@mickeyn mickeyn too early for 5.24
05784c2
@oalders oalders Stop testing on Perls < 5.22.
dbac965
@oalders oalders Merge pull request #488 from CPAN-API/mickey/perl_5_22
upgrade Perl to 5.22
2f37a82
@coveralls

Coverage Status

Coverage increased (+1.4%) to 90.24% when pulling 2f37a82 on mi/es2 into 480d931 on master.

mickeyn and others added some commits Jun 2, 2016
@mickeyn mickeyn Support queueing from Script::Release
79cbca8
@mickeyn mickeyn add missing 'version_numified' to the index 1865b0f
@mickeyn mickeyn remove internal attribute 'section' from the index
db0a8bf
@oalders oalders Merge pull request #489 from CPAN-API/mickey/queue_from_script_release
Support queueing from Script::Release
37f52c6
@coveralls

Coverage Status

Coverage increased (+1.4%) to 90.242% when pulling 37f52c6 on mi/es2 into 480d931 on master.

@oalders oalders Merge pull request #490 from CPAN-API/mickey/update_mapping
update mapping
332c68b
@coveralls

Coverage Status

Coverage increased (+1.5%) to 90.292% when pulling 332c68b on mi/es2 into 480d931 on master.

@mickeyn mickeyn fix missing module use (broken queue script)
dab77b6
@coveralls

Coverage Status

Coverage increased (+1.5%) to 90.294% when pulling dab77b6 on mi/es2 into 480d931 on master.

mickeyn and others added some commits Jun 6, 2016
@mickeyn mickeyn remove gid & uid from file stat
4841942
@oalders oalders Merge pull request #492 from CPAN-API/mickey/removed_stat_gid_uid
remove gid & uid from file stat
af7850c
@mickeyn mickeyn fix 'undefined' warning
6d28864
@coveralls

Coverage Status

Coverage increased (+1.5%) to 90.294% when pulling 6d28864 on mi/es2 into 480d931 on master.

@coveralls

Coverage Status

Coverage increased (+1.5%) to 90.296% when pulling 6d28864 on mi/es2 into 480d931 on master.

mickeyn and others added some commits Jun 9, 2016
@mickeyn mickeyn script/latest: support 'force update'
049b797
@oalders oalders Merge pull request #495 from CPAN-API/mickey/latest_add_force_flag
script/latest: support 'force update'
b74fc5b
@coveralls

Coverage Status

Coverage increased (+1.5%) to 90.294% when pulling b74fc5b on mi/es2 into 480d931 on master.

@mickeyn mickeyn script/river: use Cpanel::JSON::XS
26c487f
@coveralls

Coverage Status

Coverage increased (+1.5%) to 90.294% when pulling 26c487f on mi/es2 into 480d931 on master.

@mickeyn
Contributor
mickeyn commented Jun 14, 2016

time to merge to master?

@oalders
Member
oalders commented Jun 14, 2016

I think so!

@oalders
Member
oalders commented Jun 14, 2016

@mickeyn would you like to do the honours?

@mickeyn mickeyn merged commit b67d768 into master Jun 14, 2016

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
coverage/coveralls Coverage remained the same at 90.294%
Details
@mickeyn
Contributor
mickeyn commented Jun 14, 2016

sure :)

@mickeyn mickeyn deleted the mi/es2 branch Jun 17, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment