Results Page Sort Newest First not showing all 264 field dates

[nmswanson]

EIU- I think the problem has to do with the value in 264, second indicator. New books with 264 _1 sort properly when we ask for Newest to Oldest, but our theses are coded with 264 _0 (date of production rather than publication) and New VuFind seems not to use this data. Old VuFind works as expected, and I think it does not use the 264 second indicator as a criterion. I can see that if multiple 264 fields were used with different dates, that would lead to confusion of which date to use in sorting. Can fixed field Date1 be used in some way? Or a hierarchy of second indicator values established for sorting?

When using our go-to search in ref for locating EIU theses: [dept name] + eastern Illinois + thesis,
sorting by Newest first in new vufind puts 2013 on top - these theses use field 260 for year published. For bio, results are here https://i-share.carli.illinois.edu/vf-eiu/Search/Results?lookfor=biological+sciences+eastern+illinois+thesis&type=AllFields&sort=year. In old vufind, 2018 is on top https://vufind.carli.illinois.edu/vf-eiu/Search/Home?lookfor=biological+sciences+eastern+illinois+thesis&type=all&start_over=1&submit=Find&sort=year - these theses use field 264 for year produced.

RT 101046

[nmswanson]

Jessica confirmed that publishDate and publishDateSort currently only look at 260 (any), 264 _1, and 264 _4.

VF 0.6 looked at Date 1 (008 bytes 07-10).

[nmswanson]

If Date1 can't be used in the logic, EIU recommends improving the 264 second indicator logic to pull dates in the following order: 1,4,0,2,3

[gibsonjc]

NBY - When using New VuFind I noticed an issue with retrieving some materials that were published before 1500.

I searched for "Terence" as author in the Basic search and excluded HathiTrust, Electronic and Microforms. I selected "Terence" from the author facet and sorted by oldest first. Nothing earlier than 1501 appeared at the beginning of the list. Although one manuscript with a questionable pre-1500 date range was retrieved, it does not appear at the top of the list. It only appears as a Manuscript format facet.

For Terence as author we have 17 pre-1500 editions and 1 with him as an added entry.

Using the Advance Search and specifying a date range for 1400-1500 also does not retrieve what we have for Terence in our collection.

I have also tried searching for all pre-1500 material in our collection using the date limit in Advanced search, again excluding HathiTrust, Electronic and Microfom. I retrieve only 201 items instead of the 3500-4000 items that we have. I sampled for some how dates are recorded in the fixed field portion of the MARC record and can't find any errors there.

I searched for "Terence" as author in the Basic search and excluded HathiTrust, Electronic and Microforms. I selected "Terence" from the author facet and sorted by oldest first. Nothing earlier than 1501 appeared at the beginning of the list. Although one manuscript with a questionable pre-1500 date range was retrieved, it does not appear at the top of the list. It only appears as a Manuscript format facet.

https://i-share.carli.illinois.edu/vf-nby/Search/Results?filter%5B%5D=-institutions%3A%22HAT%22&filter%5B%5D=-format%3A%22Electronic%22&filter%5B%5D=-format%3A%22Microform%22&filter%5B%5D=author_facet%3A%22Terence%22&lookfor=Terence&type=Author&sort=year+asc

Using the Advance Search and specifying a date range for 1400-1500 also does not retrieve what we have for Terence in our collection.

https://i-share.carli.illinois.edu/vf-nby/Search/Results?sort=year+asc&join=AND&lookfor0%5B%5D=Terence&type0%5B%5D=Author&bool0%5B%5D=AND&filter%5B%5D=-institutions%3AHAT&filter%5B%5D=-format%3AElectronic&filter%5B%5D=-format%3AMicroform&illustration=-1&daterange%5B%5D=publishDate&publishDatefrom=1400&publishDateto=1500&filter%5B%5D=author_facet%3A%22Terence%22

I have also tried searching for all pre-1500 material in our collection using the date limit in Advanced search, again excluding HathiTrust, Electronic and Microfom.

https://i-share.carli.illinois.edu/vf-nby/Search/Results?sort=year+asc&join=AND&lookfor0%5B%5D=&type0%5B%5D=AllFields&lookfor0%5B%5D=&type0%5B%5D=AllFields&lookfor0%5B%5D=&type0%5B%5D=AllFields&bool0%5B%5D=AND&filter%5B%5D=-institutions%3AHAT&filter%5B%5D=-format%3AElectronic&filter%5B%5D=-format%3AMicroform&illustration=-1&daterange%5B%5D=publishDate&publishDatefrom=&publishDateto=1500

Here are a few bib ids that should have been retrieved in all three of these searches
713673
713660
713642
713636
713616

RT 102725

[gibsonjc]

Early testing on DEVEL shows that using the getPublicationDate function for publishDate and publishDateSort (instead of the default getDates and getFirstDate, respectively) will improve some of these situations.

[gibsonjc]

Reindexing Phase 3 performed on 13May2019 now uses the 008 date and new indicator definitions for all scenarios of 260/264 data. This greatly improves sorting.

Indicator data is also now used for display placement in the record and to customize the labels:

Label | MARC tag | First Indicator | Second Indicator | Subfields to display | Core or Description Tab?
Published | 260 | Blank | any | abc | Core
Published | 264 | Blank | 1 | abc | Core
Published (Most Recently) | 260 | 3 | any | abc | Core
Published (Most Recently) | 264 | 3 | 1 | abc | Core
Published (Formerly) | 260 | 2 | any | abc | Description Tab
Published (Formerly) | 264 | 2 | 1 | abc | Description Tab
Produced | 264 | Blank, 2, 3 | 0 | abc | Description Tab
Distributed | 264 | Blank, 2, 3 | 2 | abc | Description Tab
Manufactured | 264 | Blank, 2, 3 | 3 | abc | Description Tab
Copyright | 264 | Blank, 2, 3 | 4 | abc | Description Tab

Closing this Issue.